=Paper=
{{Paper
|id=Vol-3889/paper6
|storemode=property
|title=Results of GRAMS+ at SemTab 2024
|pdfUrl=https://ceur-ws.org/Vol-3889/paper6.pdf
|volume=Vol-3889
|authors=Binh Vu,Craig Knoblock,Fandel Lin
|dblpUrl=https://dblp.org/rec/conf/semtab/VuKL24
}}
==Results of GRAMS+ at SemTab 2024==
https://ceur-ws.org/Vol-3889/paper6.pdf
Results of GRAMS+ at SemTab 2024
Binh Vu1,∗ , Craig A. Knoblock1 and Fandel Lin1
1
USC Information Sciences Institute, Marina del Rey, CA 90292, USA
Abstract
There is an enormous number of tables available on the Web. However, it is difficult to automatically use the
tables in data analytic pipelines because of the lack of semantic understanding of their structure and meaning. To
address this problem, our approach, GRAMS+, automatically creates semantic descriptions of tables using distant
supervision. SemTab is an annual challenge that provides a diverse set of benchmarks for systems that match
tabular data with knowledge graphs. In this paper, we present the results of GRAMS+ at SemTab 2024 in the
Accuracy Track. The results show that GRAMS+ is scalable and achieves competitive performance in the tasks in
which we participated.
Keywords
SemTab 2024, Semantic Description, Semantic Table Interpretation, Knowledge Graphs, Semantic Web, Data
Integration
1. Introduction
Matching tabular data to an ontology or a knowledge graph is an essential problem in Data Integration.
The task is to annotate types of columns in the tables using classes of the target ontology and relations
between columns using the ontology properties. We developed a novel approach, GRAMS+ [1], ad-
dressing this problem using distant supervision. The approach leverages the fact that some data in a
table will often overlap with data in a knowledge graph (KG), which can be used to discover candidate
types and relationships in the table. Then, the approach uses two neural networks (NN) trained with a
labeled dataset generated automatically from Wikipedia tables to predict the final column types and
relationships.
The Semantic Web Challenge on Tabular Data to Knowledge Graph Matching (SemTab) is an annual
challenge with the goal of providing benchmarks and evaluations of existing solutions to this problem.
In this paper, we present the results of GRAMS+ at the SemTab 2024 challenge focusing on the Accuracy
Track. Our approach successfully annotates a very large number of tables and achieves first place on
the tasks in which we participated.
2. The SemTab Challenge
The SemTab 2024 challenge consists of several tracks ranging from semantic table interpretation to
dataset assessment and contributions. We focus on the Accuracy Track, which is relevant to our
approach. This track contains four matching tasks: (1) the Cell Entity Annotation (CEA) matches a cell
to a KG entity, (2) the Column Type Annotation (CTA) assigns a KG class to a column, (3) the Column
Property Annotation (CPA) assigns a KG property to the relationship between two columns, and (4)
Topic Detection (TD) assigns a KG class to a table. Figure 1 shows an example table annotation.
There are two types of tables in this track: horizontal tables (or relational tables) and entity tables.
A horizontal table is a grid where each row represents an entity and each column shares the same
semantic type (e.g., Figure 1). An entity table describes a single entity, where each row contains a
property of that entity.
SemTab’24: Semantic Web Challenge on Tabular Data to Knowledge Graph Matching 2024, co-located with the 23rd International
Semantic Web Conference (ISWC), November 11-15, 2024, Baltimore, USA
Envelope-Open binhvu@isi.edu (B. Vu); knoblock@isi.edu (C. A. Knoblock); fandel.lin@usc.edu (F. Lin)
GLOBE https://binh-vu.github.io/ (B. Vu)
Orcid 0000-0001-5808-9288 (B. Vu); 0000-0002-6371-4807 (C. A. Knoblock); 0000-0001-7024-2476 (F. Lin)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Figure 1: An example of a table with annotation
Finally, the standard micro precision, recall, and F1 are used to measure the performance of the
participating systems [2].
3. GRAMS+ Approach
Figure 2 shows the overall approach of GRAMS+. It starts by finding KG entities that are mentioned
in a table. Then, we use a neural network (NN) to compute the scores of candidate entities of each
table cell. The NN model is trained with a labeled dataset automatically generated from Wikipedia
tables. Using the discovered candidates and their scores, we predict column types (CTA) and column
relationships (CPA).
Figure 2: Overall approach of GRAMS+
We generate the labeled dataset by leveraging the hyperlinks inside the Wikipedia tables to find
corresponding Wikidata entities and predict columns’ relation- ships based on the linked entities. We
remove context-inconsistent hyperlinks by first automatically assigning a type to each column based
on the most common type of its entities. Then, we employ a blocklist to remove all links in a column
if the column header is incompatible with the predicted column types. The blocklist is constructed
by manually verifying headers that appeared in multiple predicted types. As our approach is detailed
in [1], the remainder of this section provides a brief overview of each component in GRAMS+, along
with any changes to fit the SemTab 2024 challenge.
3.1. Entity Linking
Following typical entity linking (EL) systems, our EL approach consists of three main steps: (1) detect
the entity columns, which are the cells that will be linked; (2) retrieve candidate entities for each cell;
and (3) compute the candidates’ likelihood.
For step 1, we directly use the target entity columns provided in SemTab’s datasets instead of running
the entity detection. To retrieve candidate entities, GRAMS+ combines multiple search strategies such
as using public Wikidata Search API, keyword search using ElasticSearch, and fuzzy search using
SymSpell. Given the huge number of tables in the Wikidata Tables dataset in Round 2 (78,745 tables),
we cannot use the public Wikidata API to search and only use the two later strategies.
To compute the candidates’ likelihood, we use a two-hidden-layer perceptron with RELU activations.
It is trained using the auto-label dataset with the following groups of features:
Surface Features include four string similarity functions between a cell and an entity name: Leven-
shtein, Jaro-Winkler, Monge Elkan, and Generic Jaccard.
Entity-context Similarity Features capture the coherence between a candidate and the surrounding
context of a cell. GRAMS+ uses two context similarity features: the weighted dot product of the column
header and the candidate description, and the number of cells matched with the candidate’s property
divided by a large constant representing the maximum number of columns in a table (e.g., 20) for
rescaling. The embeddings are computed from a Sentence Transformer model [3]1 , and the weights of
embedding dimensions are learnable parameters. Note that GRAMS+ trains two entity linking models
for tables with and without headers. Because tables from the SemTab datasets do not have column
headers, GRAMS+ uses the model trained on tables without headers.
Entity Prior Features bias the predictions toward popular entities. Currently, we use the normalized
log page rank of a candidate as the prior feature. The normalized log page rank of an entity 𝑒 is calculated
as follows:
log(pagerank(𝑒)) − min𝑒 ′ ∈ℰ log(pagerank(𝑒 ′ ))
max𝑒 ′ ∈ℰ log(pagerank(𝑒 ′ )) − min𝑒 ′ ∈ℰ log(pagerank(𝑒 ′ ))
where ℰ is the set of entities in KG, pagerank(𝑒) is the pagerank of an entity 𝑒.
3.2. Column Type Prediction
To predict the type of a column, we use a greedy algorithm that first selects the type with the highest
score from the set of types directly found in the candidate entities of a column. Then, it iteratively
refines the prediction by replacing it with an ancestor type within 𝑑 distance of the directed types if
the score difference is larger than a specific threshold 𝛿 until 𝑑 reaches the maximum chosen distance
(max_distance). The score of a type is computed by summing the maximum likelihood of the candidate
entities of the type for each cell and then dividing by the number of rows. We use the same threshold (𝛿
= 0.1) and maximum distance (max_distance = 2) as in [1].
3.3. Column Relationship Prediction
To predict the relationship of a column, GRAMS+ first constructs a candidate graph containing potential
relationships between columns. Then, GRAMS+ uses a classifier to predict the likelihood of each link
in the graph. As the SemTab challenge provides pairs of target columns for predictions, we directly use
the most likely relationships between target columns as the final predictions.
The classifier employed to predict the likelihood of links is also a two-hidden-layer perceptron with
RELU activations. It is trained on the auto-label dataset with features such as the relative frequency of
discovering the link from top K entities, the average link likelihood, the relative frequency of finding
contradicting information between the table data and KG data, and whether there is a many-to-many
relationship between the source and target of the link.
1
We use the pretrained all-mpnet-base-v2 model.
Table 1
Performance of GRAMS+ on CPA and CTA tasks. Precision and F1 scores are reported in percentage
CPA CTA
Dataset
F1 Precision Recall Rank F1 Precision Recall Rank
Wikidata Tables round 1 89.8 98.8 82.30 1 92.9 92.9 92.9 1
Wikidata Tables round 2 89.9 99.2 82.19 1 95.6 95.6 95.6 1
4. SemTab 2024 Results
Table 1 reports the performance of GRAMS+ on the Wikidata Tables datasets. We cannot run GRAMS+
on the tBiodiv and tBiomed datasets because the values of the subject columns’, which contain the main
entities, were anonymized. Since the names are changed, these datasets focus on a different aspect of
the problem, which is identifying the anonymous entities. This is not the focus of GRAMS+, and we
leave it for future work.
At the time of writing the paper, GRAMS+ achieves first place among the participants on the Wikidata
Tables datasets. The two datasets, in total, have approximately 109,000 tables. This shows that GRAMS+
is scalable and can handle a large number of tables.
5. Related Work
Table Understanding is an essential problem in Data Integration and has attracted many studies over
the years. A comprehensive related work to GRAMS+ can be found in [1]. In this section, we briefly
discuss work related to GRAMS+ in the setting of the SemTab challenge.
Most systems participating in the SemTab, including GRAMS+, exploit the existing knowledge in
a KG. Typically, they first identify KG entities in a table (CEA) and match the properties of entities
with values in the table to find column types (CTA) and relationships between columns (CPA). The best
performing systems in SemTab such as MTab [4], DAGOBAH [5], and others such as KGCode-Tab [6],
LinkingPark [7], BBW [8], TorchicTab-Heuristic [9], and SemTex [10] improve various aspects of the
pipeline such as candidate entity retrieval, scoring functions to rank the matched results, or repeat
the pipeline several times or until reaching equilibrium. Compared to GRAMS+, they often rely on
hand-crafted scoring functions, while GRAMS+ uses distant supervision to learn to classify correct
entities and column relationships. Moreover, GRAMS+ tackles a general setting where we need n-ary
relationships to correctly model data in the tables.
The SemTab 2023 and 2024 also include other tasks, such as Table Topic Detection and Matching
Table Metadata to KG. These are not the focus problems of GRAMS+, and we leave them for future
work.
6. Conclusion
This paper presents the results of GRAMS+, a distant supervised approach for annotating column types
and relationships of tables, for the SemTab 2024 Accuracy Track. GRAMS+ achieves rank 1 for datasets
on which it was evaluated.
In future work, we plan to improve the performance of GRAMS+ by jointly predicting column types
and relationships. We also plan to extend GRAMS+ to leverage table context, metadata, and modeling
instructions to support tables without overlapping data to a target knowledge graph.
Acknowledgements
This material is based upon research supported by the Defense Advanced Research Projects Agency
(DARPA) under Agreement No. HR00112390132 and Contract No. 140D0423C0093. Any opinions,
findings and conclusions or recommendations expressed in this material are those of the authors and
do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA); or its
Contracting Agent, the U.S. Department of the Interior, Interior Business Center, Acquisition Services
Directorate, Division V.
References
[1] B. Vu, C. A. Knoblock, B. Shbita, F. Lin, Exploiting distant supervision to learn semantic descriptions
of tables with overlapping data, in: The Semantic Web–ISWC 2024: 23th International Semantic
Web Conference, ISWC 2024, November 11–15, 2024, Proceedings 20, Springer International
Publishing, 2024.
[2] O. Hassanzadeh, N. Abdelmageed, V. Efthymiou, J. Chen, V. Cutrona, M. Hulsebos, E. Jiménez-Ruiz,
A. Khatiwada, K. Korini, B. Kruit, et al., Results of semtab 2023, in: CEUR Workshop Proceedings,
volume 3557, 2023, pp. 1–14.
[3] N. Reimers, I. Gurevych, Sentence-BERT: Sentence embeddings using siamese BERT-Networks
(2019). arXiv:1908.10084 .
[4] P. Nguyen, I. Yamada, N. Kertkeidkachorn, R. Ichise, H. Takeda, SemTab 2021: Tabular data
annotation with MTab tool, http://ceur-ws.org/Vol-3103/paper8.pdf, ???? Accessed: 2023-10-6.
[5] V.-P. Huynh, Y. Chabot, T. Labbé, J. Liu, R. Troncy, From heuristics to language models: A journey
through the universe of semantic table interpretation with DAGOBAH, Semantic Web Challenge
on Tabular Data to Knowledge Graph Matching (SemTab) (2022).
[6] X. Li, S. Wang, W. Zhou, G. Zhang, C. Jiang, T. Hong, P. Wang, KGCODE-Tab results for SemTab
2022, https://ceur-ws.org/Vol-3320/paper5.pdf, ???? Accessed: 2023-10-6.
[7] S. Chen, A. Karaoglu, C. Negreanu, T. Ma, J.-G. Yao, J. Williams, A. Gordon, C.-Y. Lin, LinkingPark:
An integrated approach for semantic table interpretation, http://ceur-ws.org/Vol-2775/paper7.pdf,
???? Accessed: 2023-10-6.
[8] R. Shigapov, P. Zumstein, J. Kamlah, L. Oberlander, J. Mechnich, I. Schumm, bbw: Matching CSV to
wikidata via meta-lookup, https://madoc.bib.uni-mannheim.de/57386/3/paper2.pdf, ???? Accessed:
2023-10-6.
[9] I. Dasoulas, D. Yang, X. Duan, A. Dimou, TorchicTab: Semantic Table Annotation with Wikidata
and Language Models, in: CEUR Workshop Proceedings, 2023, pp. 21–37.
[10] E. G. Henriksen, A. M. Khorsid, E. Nielsen, A. M. Stück, A. S. Sørensen, O. Pelgrin, Semtex: A
hybrid approach for semantic table interpretation, 2023.