A large portion of structured data does not yet reap the bene ts of the Semantic Web. Therefore, The \Tabular Data to Knowledge Graph Matching" competition at ISWC tries to bridge this gap by evaluating and promoting the creation of such semantic annotations tools. Besides annotating data semantically, the system should also be able to further augment the datasets based on the provided annotations. In this paper, we propose a system that is capable of both annotating and augmenting a dataset by using the interpretable embedding technique INK. The \Tabular Data to Knowledge Graph Matching" competition was used to evaluate the proposed annotation capabilities of our proposed system.
DBpedia or Wikidata column type (Column-Type Annotation (CTA)), (ii) a
DBpedia or Wikidata entity has to be assigned to the di erent cells (Cell-Entity
Annotation (CEA)), and (iii) relations between di erent columns have to be
inferred, when possible (Columns-Property Annotation (CPA)). In our example
Figure 1, the CEA task would have to assign the dbr:Coldplay or wiki:Q45188
to text description of Coldplay in Col0. For the CTA task, Col3 must be
assigned to dbo:PersonFunction or wiki:Q66715801. The relation between Col1
and Col2 for the CPA task must be de ned as dbp:birthDate or wiki:P569 in a
semantic annotation system. The competition consists of di erent rounds, with
structured les from di erent domains. No ground truth labels are provided
upfront, which means only unsupervised learning methods can be used.
Most of the existing solutions are based on external lookup methods and infer
the column and property types afterwards by either clustering the entities based
on an uninterpretable embedded vector or by using entity-speci c scoring
procedures [
CPA
CTA
Col0 Col1 Col2 The Killers Brandon Flowers 21 June 1981 Singer Coldplay CEA Chris Martin 2 March 1977 Singer U2 The Edge 8 August 1961 Guitarist ABBA Benny Andersson 16 December 1946 Composer Col3
Inception 2001 + Augment 1996 1976 1972
Therefore, in this paper, we describe MAGIC: a data mining tool to augment
a structured le with data residing in a KG. In order to mine this augmented
data, MAGIC will make use of INK [
We have organised the rest of our paper as follows: In Section 2, we give a general introduction to our interpretable embedding technique INK. Next, Section 3 describes how INK is being used in our semantic annotation tool MAGIC. Implementation details are provided in Section 4. Section 5 shows the results obtained during the \Tabular Data to Knowledge Graph Matching" competition. To illustrate the additional bene ts of both the interpretable embedding technique INK in combination with the MAGIC platform, we demonstrate how an existing structured le can be augmented with additional information within a KG in Section 6. Finally, Section 7 concludes this paper and shows some additional future research directions.
Various techniques exist to transform information residing in KGs to a more
appropriate format for an ML model, such as RDF2Vec [
Singer
UniversityColegeLondon
Will Champion
Date_of_birth Occupation
31July1978 Drummer
1996
resentation can be used to perform some more general machine learning tasks,
such as node classi cation. More information about INK and the performance
of this technique on several node classi cation tasks can be found in previous
work [
The interpretable embedding approach as discussed in Section 2 is used in a system called MAGIC to annotate structured les. An overview of the MAGIC system is provided in Figure 3. In general, three modules can be derived: A preprocessing module to select the appropriate annotation candidates, a processing module based on INK to get the interpretable embedding and select the best matches and at last, a post-processing module to o oad the obtained information to the corresponding competition tasks. The rest of the section describes, in detail, the eight di erent steps to process the structured input le.
TCAUhoB2elBdCAKpoillla0eyrs TCBBhhreaenrninsEdydMogCAnaeonrFldt1lioenrwsseorsn 822116AMJDuuagenrucceeshCtm1o119bl9982e761r711946 SSCGiiounnmiggteCaeprrrooilss3ter
Structured File 1
3 Candidate Selection 2 5 ESmebleecdtdioinng 6 Entity Propagation CEA
Magic 7 8
CPA Property Propagation Column Selection
CTA
De ning the Major Column in a Structured File In most cases, one single column holds multiple relationships to other columns within a table, for which annotation should be provided. This column can be seen as the major column. In our example of Figure 1, Col1 is our major column as both the information of Col0, Col2 and Col3 are derived from the information in this column. In many cases, this major column is known upfront and can be provided as additional input. During the \Tabular Data to Knowledge Graph Matching" competition, the major column was derived from the CPA target le when available. If such a target le was not provided, the MAGIC approach was rerun for every available column. 3.2
External Entity Lookup
Each cell description of only the major column is then provided to the more
general MAGIC annotator. A pool of possible annotation candidates is generated
from these cell descriptions. For the competition tasks which requires DBpedia
annotations, the DBpedia Spotlight [
Candidate Selection The external entity lookup services are not restricted, which means all possible candidates matching the provided cell description are returned. Based on multiple prede ned characteristics, candidates can be pre ltered. The candidate selection step reduces, in this perspective, the number of matches obtained from the external API's. In this rst version of the MAGIC system, we neglected any intelligent ltering of candidates. 3.4
Generating embeddings with HDT backend
For the selected candidate annotations, an INK embedding of depth 2 was
generated as discussed in Section 2. To limit the number of external SPARQL requests,
INK extracted the neighbourhood information for these embeddings from a local
compact data structure called (Header, Dictionary, Triples) or HDT [
{ Header: The Header holds metadata describing an HDT semantic dataset
using plain RDF. It acts as an entry point and shows the key properties of
the content even before retrieving the whole dataset.
{ Dictionary: The Dictionary is a catalogue comprising all the di erent terms
used in the dataset, such as URIs, literals and blank nodes. A unique
identier (ID) is assigned to each term, enabling triples to be represented as tuples
of three IDs, which reference their respective subject/predicate/object term
from the dictionary.
{ Triples: As stated before, the RDF triples can now be seen as tuples of three
IDs. Therefore, the Triples section models the graph of relationships among
the dataset terms. By understanding the typical properties of RDF graphs,
we can come up with more e cient ways of representing this information,
both to reduce the overall size, but also to provide e cient search operations.
Popular and well-known libraries (like RDFLib [
Selecting the best candidate embedding At this point, interpretable embeddings of depth 2 were generated for multiple candidates, originating from a single cell within the major column of our structured le. Within these interpretable embeddings, a next function will search for matching information residing in the other cells on the same row from which the candidate embeddings were generated. In our example, the text description of Chris Martin can return two Wikidata entities: one is the Coldplay singer (Q712860), the other candidate is an American football player (Q519982). The INK embedding generated for the Coldplay singer will contain the information residing in the other cells of that same row (the is part, birth date and occupation relation). As an embedding of depth 2 is provided, the labels of these associated nodes are incorporated in the embedding. The best candidate or embedding can be easily selected by counting the number of times additional information residing in other cells from the same row, is also available in the generated embedding of the candidate. In our example, the Chris Martin Coldplay singer embedding will have a matching count of 3, while the other Chris Martin has a zero match count. Therefore, the Chris Martin Coldplay singer annotation is selected as the best candidate in this situation. 3.6
Filling Additional Cells based on Selected Candidates Selecting the best candidate according to the embedding also provides all information to the cells within the same row of the original structured le. Instead of
performing an entity lookup for those cells, the annotations are directly derived from the major column embeddings. This eventually reduces both the costs of performed SPARQL lookups and external API calls. The annotations are all stored within a dictionary and output for the CEA task. 3.7
De ning Relationships Between Cells Similarly, the embedding also provides the information going from the major column cell to the neighbouring cells within the same row. This information is directly derived from the provided embedding by selecting the object or data property relationships from the INK embedding. The combination of a relationship between two cells is kept within a dictionary. After the procedure to provide annotations for all cells is nished, the relationships between all those two cells are counted and the relationship with the maximal count is returned for the CPA task. 3.8
De ning column types based on additional type embeddings The previous process is iterated for each cell within our major column. After all those annotations are provided, the column type can be derived from all cells containing annotated values. For all cells, an INK embedding of depth 1 is generated and the rdf:type for DBpedia and Wikidata P31 relationships are kept to determine the column type annotations. Again, the annotation with the highest count value is kept and returned for the CTA task.
Both MAGIC3 and INK4 are implemented in Python and are made available on Github for future research. INK's implementation details can be found in the original paper. The MAGIC code is designed to perform evaluations for both DBpedia and Wikidata tasks but can be adapted to any other task. { The code to select the major column is currently left outside MAGIC. This makes it possible for users to provide either this major column by itself or to write speci c code to determine this column within a structured le. { The code to search for entity candidates is abstracted. This ensures that future approaches can integrate other entity search API's (such as the DBpedia lookup service) without redesigning the internal MAGIC code. Also, additional preprocessing steps, such as translations, spell checks, etc. can be added to this component in the future. { INK abstracts which data source it uses to generate the embeddings. In this version of the system, the embeddings were generated using an HDT backend and an HDT INK connector was made based on RDFlib to perform
this task. In future projects, other already existing connectors (such as the INK Stardog5 connector to connect to a triple store) can be used or created when needed. 5
The \Tabular Data to Knowledge Graph Matching" competition of 2021 consisted of three rounds in which multiple structured CSV les had to be annotated for either one, two or all CEA, CPA and CTA tasks. The di erent rounds consisted of multiple datasets from various domains where either DBpedia or Wikidata annotations were requested.
In total, four di erent metrics were used to evaluate the system. On the one hand, we had the F1-score, which is a harmonic mean of the precision and recall of our system: precision = recall =
F1 = #correct annotations #annotations made #correct annotations
jtarget cellsj 2 precision recall
precision + recall During the second round, new metrics were used for the CTA challenge. Perfect annotation were encouraged, and at same time one of its ancestors (okay annotation) were also evaluated. Thus we calculate Approximate Precision (APrecision), Approximate Recall (ARecall), and Approximate F1 Score (AF1).
APrecision = Pa2aallllanannnotoattaitoinonss#g(a)
AAFR1ec=all2 A=APPrPercecicsoiilsoi2nona+ll tAaArRgReetceaacclolallllultmanrgse(tmcaoxluamnnnsot#ation score(col)) with # denotes the number, g(a) returns the full score 1.0 if a is a perfect annotation, returns 0:8d(a) if a is an ancestor of the perfect annotation and its depth to the perfect annotation d(a) is not larger than 5, returns 0:7d(a) if a is a descendent of the perfect annotation and its depth to the perfect annotation d(a) is not larger than 3 and returns 0 otherwise. max annotation score(col) returns g(a) if col has an annotation a, and 0 if col has no annotation. All evaluations were performed on the same 32 core Intel(R) Xeon(R) CPU E52650 v2 @ 2.60GHz cluster node with 125 gigabyte RAM. The Wikdata HDT datasource of 3 march 2020 was used for all Wikidata related tasks. The October 2016 English DBpedia HDT datasource was used for all DBpedia related tasks. The results of our approach are summarized in Table 1 respectively.
(1) (2)
The MAGIC system has competitive results when a clear CPA task has been de ned. When such a task and corresponding links between the columns within our structured les is not de ned, the system has more di culties selecting the correct entity descriptions. For some datasets, our system extracted too many candidate embeddings to evaluate. This resulted in memory issues and loss of information within the corresponding tasks. The evaluations of these datasets (in particular, the Wikidata tables and GitTables) are therefore not provided in this table. 6
Besides the provided annotations for the CEA, CPA and CTA task within the \Tabular Data to Knowledge Graph Matching" competition, the MAGIC framework holds an additional advantage compared to its competitors. The INK embeddings generated in the magic system are interpretable and used to match the nearby cells within the same row. Additionally, information originating provided from a new relationship, which holds for all cells within a column can be added when it is available from the interpretable embedding. When e.g., we annotated all cells within our structured le in Figure 1. Combining all INK embeddings of Col0 will reveal additional information regarding the Bands listed in this le. One such additional relationship could be the Inception year, which is information that can easily be added as a new column to our original dataset. MAGIC can automate this process, revealing new possibilities to extend the original dataset with new information.
To make these bene ts even more tangible, an additional GUI application has been developed that displays both the annotation and augmentation parts. An example of the GUI is visualized in Figure 4. A video about this GUI application, using the basic example of Figure 1, is also made availabe6 7
In this paper, a system to annotate and augment a structured le with semantic knowledge is being proposed. This system shows the bene ts of combining the interpretable embedding technique INK with a semantic annotation tool. Future work can now focus on both the preprocessing and post-processing functionalities to improve the generated annotations. Currently, matches are provided on exact string comparisons, without taking any malformed or misspelt text into account. The generated embeddings are also not used to detect wrongly annotated cells. Simple outlier detection or clustering tools based on the generated embedding of a single column can already help to lter those wrong annotations. At last, a thorough evaluation is needed of how this system can help to augment existing datasets and how this augmented data can help in more broad ML tasks. 8
Bram Steenwinckel (1SA0219N) is funded by a strategic base research grant of the Fund for Scienti c Research Flanders (FWO). 6 https://www.youtube.com/watch?v=ZhTKxcTBZNE