In this paper, we present the latest improvements of the DAGOBAH system that performs automatic pre-processing and semantic interpretation of tables. In particular, we report promising results obtained in the SemTab 2021 challenge thanks to optimisations in lookup mechanisms and new techniques for studying the context of nodes in the target knowledge graph. We also present the deployment of DAGOBAH algorithms within the Orange company via the TableAnnotation API and a front-end DAGOBAH user interface. These two access methods enable to accelerate the adoption of Semantic Table Interpretation solutions within the company to meet industrial needs.
Tables constitute a major source of knowledge since large parts of companies
internal repositories and Web pages are represented in tabular formats. Hence,
there is a strong interest in Semantic Table Interpretation (STI), i.e., methods
that automatically interpret tables with the support of a knowledge graph (KG)
by associating each cell with an individual (Cell-Entity Annotation, CEA), each
column with a class (Column-Type Annotation, CTA), and each pair of columns
with a property (Columns-Property Annotation, CPA) [
The DAGOBAH algorithms, conjointly developed by Orange and
EURECOM, have been evaluated on all editions of the SemTab challenge and they
have now reached a maturity level that allows to address industrial use cases
related to STI and inherently present within Orange. Indeed, Orange is a
multinational company operating in a wide range of business domains (e.g.,
telecommunications, banking, multimedia content, cybersecurity). Consequently, Orange
produces a high volume of heterogeneous tabular data. Thanks to STI
techniques, these data could be strategically leveraged, e.g., by structuring dormant
knowledge, thus making it actionable and usable through Q/A engines [
Both the SemTab 2021 challenge and the aforementioned industrial needs
motivated the following improvements in the entity scoring algorithm of the
DAGOBAH SL 2020 system [
These specific improvements yield the DAGOBAH SL 2021 system which is detailed in Section 2. We report on the results of our experimental evaluation as well as the insights we gain from the challenge leaderboard in Section 3. Section 4 introduces our eforts towards the usability of DAGOBAH within Orange via the TableAnnotation RESTful API and a new front-end DAGOBAH user interface. Finally, we reflect on the SemTab challenge and the adoption of STI solutions within the enterprise in Section 5. 2
DAGOBAH provides an end-to-end process that annotates relational tables with constituents of a KG such as Wikidata. Its workflow consists of the four sequential steps depicted in Figure 1. Given a relational table as input, the preprocessing determines table metadata and annotation targets (Section 2.1). The entity lookup module then collects candidates from the KG for each target cell of the table (Section 2.2). The pre-scoring module evaluates each candidate to determine a confidence score (Section 2.3). Next, Columns-Property Annotation (CPA) and Column-Type Annotation (CTA) are carried out (Section 2.4). Finally, Cell-Entity Annotation (CEA) is performed in order to compute the final entity scores taking CTA and CPA into account (Section 2.4). 2.1
Table Pre-Processing In real use cases, annotating tables is complex because of little prior knowledge about their structure and content. Therefore, pre-processing tables can facilitate later annotations. In this view, DAGOBAH pre-processing steps generates metadata about a table via four main tasks: orientation detection, header detection, key column detection3 and column primitive typing. This primitive typing detects named entities (e.g., Location, Organization, Person), literals with
units (e.g., Distance, Speed, Temperature), or miscellaneous literals (e.g., Email,
URL, IP Address). The pre-processing step was particularly helpful for the
BioDivTab [
DBpedia entities. The lookup service collects English resources with their labels in all languages. To increase lookup coverage, labels are enriched with the values of 25 alias properties such as abbreviation, birthName, or originalTitle. In addition, labels and aliases of all redirected entities are also included.
We average the character-based and token-based edit distances5 to evaluate the similarity between a cell mention and the set of labels of each candidate 4 Since target cells are given in the SemTab challenge, this feature is not used in our experiments. 5 https://github.com/seatgeek/thefuzz entity. This helps to solve the “out of order problem” where a string can have diferent orders for its substrings ( e.g., “Elon Musk” and “Musk Elon”). 2.3
The pre-scoring step evaluates with a preliminary score the relevance of a candidate entity ec ∈ Ec(em) for a table cell em:
P Sc(ec, em) = Sccontext(Ngraph(ec), Ntable(em)) × eγ(Scsim(ec,em)−1) (1) This pre-score is the product of a context factor and a literal factor Scsim(ec, em). The latter returns the highest Levenshtein-based matching ratio between the cell and the label and aliases of the candidate. Aliases are penalized with a ratio weighted by 0.9 since we consider labels to be more important. The amplification factor γ ∈ N+ determines the importance of the textual similarity. We empirically observed that 2 was an appropriate amplification factor for the SemTab challenge.
The improvements in DAGOBAH SL 2021 mainly concern the context factor defined as follows:
Sccontext(Ngraph(ec), Ntable(em)) =
Pi wi × sni
Pi wi (2) where Ntable(em) is the set of neighboring cells in the same row as em and Ngraph(ec) is the set of neighboring nodes of ec in the KG6. For each neighboring cell ni ∈ Ntable(em), sni is its neighborhood matching score w.r.t. Ngraph(ec).
DAGOBAH SL 2021 solves two issues related to the context score calculation with respect to DAGOBAH SL 2020: Expensive evaluation. Each sni was evaluated by iterating over all context nodes in Ngraph(ec) to find the best matching node. Hence, a performance bottleneck arose when scoring a generic entity with millions of edges in the KG. For example, let’s consider the cell “Belfort” in Figure 1 and the Wikidata candidate entity Q171545. To check whether the neighboring cell “Bourgogne Franche Comté” is in the context of Q171545, we performed a comparison with each of the ∼ 1000 nodes in Ngraph(Q171545), including Q142 (France), Q3371185 (Paul Faivre), etc. (Figure 2a).
One-hop graph contexts. Ngraph(ec) consisted of nodes only one hop away from ec. Consequently, a neighboring cell ni ∈ Ntable(em) matching with a node two hops away from ec was not considered in the context of ec. For example, given the one-hop context of Q171545 (Belfort) in Figure 2a, we wrongly considered that Bourgogne Franche Comté had no relation with Belfort whereas it is the region of Territoire de Belfort (French department) whose capital is Belfort.
DAGOBAH SL 2021 improves both the efficiency and the expressiveness of the context score by avoiding the exhaustive scoring and exploiting more expressive contexts for an entity via two-hop predicate paths. 6 Neighboring nodes are connected to ec via predicate paths in the KG, regardless of the predicate direction.
matching score sni in Equation (2) indicates whether a neighboring cell ni of em matches with a neighboring node of ec. Loosely speaking, computing sni comes down to searching a candidate entity for ni in Ngraph(ec) and to assessing its similarity. In our running example, Q18578267 is a candidate for cell “Bourgogne Franche Comté” in the two-hop context Ngraph(Q171545) (Figure 2b). From this observation, we propose the following way to efficiently compute sni. The entity lookup step (Section 2.2) outputs candidate entities Ec(em) for a target cell em but also candidate entities Ec(ni) for its neighboring cells ni. Hence, we check if a candidate entity ei ∈ Ec(ni) is in Ngraph(ec). In that case, sni is simply calculated by comparing the labels of the neighboring cell ni and the matching node ei, which avoids additional comparisons with other nodes in Ngraph(ec).
To check if ei ∈ Ec(ni) is in Ngraph(ec), we actually check if ei is connected to ec by a predicate path in the KG. We chose to compute such predicate paths since they are useful in the soft context scoring. To efficiently find predicate paths between ec and ei, we extract the one-hop subgraphs Gec and Gei around ec and ei. If an intermediate node v is present in both Gec and Gei , the paths pointing to v in the two sub-graphs are concatenated. In our running example, capitalOf we find the following predicate path: Belfort −−−−−→ Territoire de Belfort −l−oc−at−e−d→In Bourgogne Franche Comté. Since we only consider one-hop subgraphs, paths can have a maximum length of two hops. This approach allows to enrich the information about an entity by including not only direct neighbors but also indirect neighbors two hops away. Such enhanced graph contexts increase the chance for a neighboring cell ni ∈ Ntable(em) to match, and thus make the context score more precise. We argue that, in the context of STI with Wikidata, paths longer than 2 hops are often noisy and meaningless, and thus can have a negative impact on the context score.
Soft Context Scoring. In Equation (2), neighborhood matching scores sni are weighted to compute the ultimate score of an entity. Indeed, each neighboring cell ni ∈ Ntable(em) contributes diferently to the annotation of the target cell em with a weight wi defined in Equation (3): wi = (3a) zs}e|i{ pd(coli) + 1 | } ({3zb) (3c)
(3d) × cznt(}c|oli{) × τz (}e|i{) .
(3) (3a) Cells containing entities should be more important than cells containing literals (e.g., date, measurement with/without unit, number) since there is a lack of literal disambiguation methods (e.g., date-time normalization, unit detection/normalization/conversion). That is why, we set sei to 1.0 if the neighboring cell ni contains an entity, and to 0.15 if ni contains a literal. (3b) A neighboring cell on the left side of the table has more chance to be a meaningful context for the target cell. Hence, d(coli) is the distance between column coli and the first object column of the table. (3c) Cells ni from a neighboring column highly connected to the target column should have a greater weight in the context. Hence, we take into account the connectivity cnt(coli) of the neighboring column w.r.t. the target column, defined as the highest occurrence of a relation potentially found between the two columns. Territoire de Belfort −l−oc−at−e−d→In
country
than Belfort −−−−→ France ←d−ip−lo−m−at−ic−R−el−at−io−n United States of America. To
quantify this, we rely on the so-called truth value τ (ei) [
1 1 + log(g(v)) (4) where g(v) is the generality of the intermediate node v, i.e., the number of its incoming and outcoming edges in the KG. Note that direct neighbors (or 1-hop paths) always get the highest truth value 1.0. 2.4
Columns-Property Annotation. The CPA task outputs the most suitable
semantic relation r for an ordered pair of columns. We adopt a majority voting
strategy that relies on the occurrence and accumulated confidence score over
rows for r. The interested reader can refer to [
Cell-Entity Annotation. The CEA task selects for a table cell em the most relevant entity among candidate entities ec ∈ Ec(em) retrieved from the KG. This step leverages both entity pre-scoring and information given by CTA and CPA to compute the final score of candidate entities. Indeed, the pre-scoring of a candidate entity ec only considers its local information, i.e., the row it belongs to. Column type output by CTA and column pair relations output by CPA allow to consider table’s global information. Hence, the final score Sc(ec, em) of a candidate entity ec is computed as follows:
Sc(ec, em) = (P Sc(ec, em) + α × scoreCT A + β × scoreCP A) 1 + α + β (5) If ec belongs to the type output by CTA for its column, then scoreCT A is equal to the score of this type, otherwise it is set to 0. In scoreCP A, we average the scores of the relations output by CPA for column pairs involving the column of ec. For each relation, if ec belongs to its domain or its range (depending on the relation orientation), then we consider the score of this relation, else it is set to 0. To strengthen (resp. weaken) a frequent (resp. unusual) CTA/CPA in the update of Sc(ec, em), a coefficient α (resp. β) is employed and defined as occurrence(CT A) (resp. occurrence(CP A) ). Note that the occurrence of CTA/CPA 2 2 is divided by 2 to prioritize the pre-scoring P Sc(ec, em). 3 3.1
To evaluate one-hop and two-hop graph contexts as well as the soft context scoring described in Section 2, we consider the four experimental settings: Setting 1 The context score of an entity is computed using only its one-hop neighboring graph. Weights wi do not follow Equation (3) but are set to 1.0 for entities and 0.15 for literals.
Setting 2 The context score of an entity is computed using its two-hop neighboring graph. Weights wi do not follow Equation (3) but are set to 1.0 for 1-hop neighbors, 0.25 for 2-hop neighbors, and 0.15 for literals.
Setting 3 The context score of an entity is computed using its two-hop neighboring graph. Weights wi follow Equation (3). This setting allows to assess if richer contexts and stricter scoring lead to better annotation. Setting 4 This setting restricts Setting 3 to 1-hop and unidirectional 2-hop predicate paths in graph contexts. This allows to evaluate the impact of bidirectional paths which are often less informative or noisy but may be helpful in some cases. 3.2
Experimental evaluation. We provide an experimental evaluation of the four aforementioned settings in Table 1. It should be noted that DAGOBAH is continuously improved. Hence, results of this evaluation are based on the current version of DAGOBAH but we also report results submitted to the SemTab challenge in gray cells for comparison. To validate the modifications proposed in Sections 2.2 and 2.3, we include the scores of the DAGOBAH 2020 system on tables annotated with Wikidata in Round 1. Submission Settings {1,2,3,4}* are similar to Settings {1,2,3,4} but slightly difer in scores and weights initialization. This does not change CEA scores but impacts CTA performances. Indeed, CTA is highly sensitive to entity scores and taxonomy weights to select the most fine-grained type among the many possible (direct and ancestor) types of entities.
DAGOBAH achieves a high performance on synthetic datasets (Round 2) whereas high-quality manually-curated datasets with complex table patterns are more difficult to annotate (Rounds 1 and 3). In HardTable, no gain is brought by using a richer graph context or a more flexible scoring. This can be explained since tables are almost fully represented in the target KG and columns can be disambiguated from their contents. On the contrary, BioTable provides remarkable ambiguities with content overlaps between columns that hinder their disambiguation (e.g., column “Gene” can be mistaken with column “Protein”). Hence, annotation seems to benefit from richer graph contexts. In BioDivTable, Setting 4 obtains the lowest scores whereas Setting 1 is comparable to Setting 3. We suppose that unidirectional 2-hop predicate paths may be noisy or not correctly considered, leading to the lowest score of Setting 4. In general, Settings 2, 3, and 4 are more precise for CEA than Setting 1. Hence, 2-hop graph contexts bring useful information. The better performance of Settings 3 and 4 compared with Setting 2 shows the efectiveness of soft context scoring. We notice that Setting 3 achieves similar performances to Setting p1 p2 4 which can be interpreted as follows. First, unidirectional paths (i.e., −→−→ and ←−←p−2) bring enough information and allow to obtain equal results comp1 pared with considering both unidirectional and bidirectional paths. Second, the country diplomaticRelation influence of noisy bidirectional paths ( e.g., Belfort −−−−→ France ←−−−−−−−−−− United States of America) is limited by the soft context scoring which prevents a degradation in the annotation quality. This allows useful bidirectional paths to contribute positively in the entity score. It can be observed that CTA and CPA performances are not as high as expected in most datasets despite CEA good performances. The development of better strategies for type and relation selection will be the subject of future works. system in Rounds 1, 2, and 3 of the SemTab 2021 challenge (in gray cells). “F1” stands for F1-score, “P” stands for Precision. Best results between settings are in bold.
Dataset
System Setting
CTA F1
P
CEA
CPA F1
P
F1 Round 1 – WDTable Round 1 – DBPTable Round 2 – BioTable Round 2 – HardTable Round 3 – BioDivTable Round 3 – HardTable
Setting 1 Setting 2 Setting 3 Setting 4 DAGOBAH 2020 Setting 2* Setting 1 Setting 2 Setting 3 Setting 4 Setting 2* Setting 1 Setting 2 Setting 3 Setting 4 Setting 4* Setting 1 Setting 2 Setting 3 Setting 4 Setting 3* Setting 1 Setting 2 Setting 3 Setting 4 Setting 4*
Setting 3* Round 3 – GitTables DBP Pre-processing + Mapping 0.07 Round 3 – GitTables SCH Pre-processing + Mapping 0.183 0.185
BioDivTab and GitTables datasets. Note that for BioDivTab and GitTables datasets, we adapted the DAGOBAH algorithms presented in this paper. Indeed, for BioDivTab, primitive types output by the pre-processing step were used to discriminate object and literal columns. A column contains literal values if its primitive type is numerical, date-time, unit, or miscellaneous. Otherwise, it is considered as an object column containing entity mentions that are passed to the lookup module. For GitTables, primitives types are manually mapped to Schema.org and DBpedia Ontology classes. 4
To quickly improve the relevance of DAGOBAH w.r.t. real use cases, we adopted a Test & Learn approach. In this view, DAGOBAH algorithms have been made available to other collaborators inside the company since the early stages of the project via the TableAnnotation API and the DAGOBAH UI.
DAGOBAH UI. This interface allows non-developers and non-AI experts to use the TableAnnotation API resources on their tables and to visualise the results in an intelligible and ergonomic form. This interface enables users to load new tables into their annotation project (Figure 4a), start the pre-processing and semantic annotation tools, and visualise the results (Figure 4b). As DAGOBAH UI is a powerful tool to demonstrate the value of STI techniques within the company or with external prospects, future developments will focus on features such as: – KG enrichment with table elements not present in the KGs; – Table enrichment with KGs by filling in missing values or adding new columns; – Interactive visualisation of the target KG, annotations resulting from CEA, CTA, and CPA, and new triples that can be generated from the table.
This front-end user interface allows collaborators to understand the interest of STI solutions. Conversely, the DAGOBAH project team can identify challenges associated with their needs, which is a valuable input for the project roadmap. Although deployment and adoption of STI methods in Orange are still in their infancy, tests on diferent use cases have been taking place for over a year via the TableAnnotation API, which has received more than 200,000 API calls. Several areas are being prioritised including entertainment (e.g., VOD catalog), data governance, and health.
(a) DAGOBAH UI allows to define projects and load tables from the local file system or pre-loaded tables from gold standards (e.g., T2D, SemTab). (b) DAGOBAH UI allows to access pre-processing and annotation details for a table. At the top, information generated by the pre-processing (e.g., orientation, header) and the cleaned table are displayed. An interactive view shows CEA, CTA and CPA annotations at the bottom. 5
The datasets provided for the SemTab 2021 challenge have extended the spectrum of difficulties for STI methods by including new target domains ( e.g., biomedical and Git data), combining KGs, or requiring schema-only annotations (Schema.org and DBpedia Ontology). These difficulties have enabled us to enhance the DAGOBAH system annotation strategies, e.g., leveraging preprocessing primitive types, and using enriched graph contexts. Nonetheless, new research directions can still be explored to embrace the heterogeneity of table types that are published on the Web: – Table structure and inner-relationships: table orientation, nested cells, layout concatenation, multi-valued cells, subjects split into several columns (e.g., ifrst and last names of a person), etc. – Out-of-KG data: entities not present in a given KG, which is often the case with companies specific data.
It is noteworthy that out-of-KG data started to be addressed in Round 3 with the GitTables dataset that required schema-only annotations from Schema.org and the DBpedia Ontology. However, this task was not fully consistent with the CTA definition adopted by the community since ground truth annotations mixed types (classes) and properties. These heterogeneous annotations may lead to inconsistent evaluations. To address the aforementioned dimensions, the DAGOBAH team is working on hard datasets that will bring new challenges to the community. 6
In this paper, we have presented the improvements made on the DAGOBAH system. With optimised lookups, richer graph contexts, and soft scoring, DAGOBAH obtained high performances during the SemTab 2021 challenge. Our future work aims at increasing the annotation accuracy for tables with non-explicit or highly ambiguous mentions. We ambition to leverage dictionaries providing abbreviations or acronyms. To ensure the genericity of our approach, such dictionaries should be built from huge amounts of documents and be applicable on various datasets. When a majority of unmatchable mentions are present in a column, embedding-based approaches could complement context scoring strategies. In this view, we believe that approaches based on language models can become a good asset in the most challenging cases.
Acknowledgments. The authors would like to thank Christophe SarthouCamy and Guillaume Jourdain for their invaluable contribution to the development of DAGOBAH UI.