SMART 2021 [1] was the second edition of the SeMantic Answer Type and Relation Prediction Task (SMART), which part of the ISWC 2021 Semantic Web Challenge. It was co-located with the 19th International Semantic Web Conference (ISWC 2020)1. The first edition SMART2020 [2] was in ISWC 2020. Given a question in natural language, the task of SMART challenge is, to predict the answer type and relations using a target ontology. The challenge had 2 tracks (answer type prediction and relation prediction) with 2 KBs, one using the DBpedia ontology and the other using Wikidata ontology. There were six submissions for answer type prediction (DBpedia) and four systems for answer type prediction (Wikidata). Similarly, there were three systems for relation prediction (DBpedia) and three systems for relation prediction (Wikidata). This volume contains peer-reviewed system description papers of all the systems that participated in the challenge. More details about the challenge can be found at https://smart-task.github.io/2021/. This challenge is focused on answer type prediction and relation prediction, which play an important role in Question Answering systems. Answer Type Prediction Given a natural language question, the task is to produce a ranked list of answer types of a given target ontology. Previous such answer type classifications in literature are performed as a short-text classification task using a set of coarse-grained types, for instance, either six types [3, 4, 5, 6] or 50 types [7] with TREC QA task2. We propose a more granular answer type classification using popular Semantic Web ontologies such as DBpedia and Wikidata. Relation Prediction Given a natural language question, the task is to identify the relation and link to the relations in KG. Depending on the number of relations in the KG, the number of relation types to be linked varies. Table 1 and Table 2 illustrates some examples. The participating systems can be either supervised (training data is provided) or unsupervised. The systems can utilise a wide range of approaches; from rule-based to neural approaches.
Eight teams competed in SMART 2021 and presented their systems at the ISWC 2021 conference. Table 3 shows their presentation titles along with the authors. 1https://iswc2021.semanticweb.org/ 2https://trec.nist.gov/data/qamain.html
For each natural language question in the test set, the participating systems are expected to provide two predictions: answer category and answer type. Answer category can be either ‘resource’, ‘literal’ or ‘boolean’. If the answer category is ‘resource’, the answer type should be an ontology class (DBpedia or Wikidata, depending on the dataset). The systems could predict a ranked list of classes from the corresponding ontology. If the answer category is ‘literal‘, the answer type can be either ‘number’, ‘date’ or ‘string’.
Category prediction will be considered as a multi-class classification problem
and accuracy score will be used as the metric. As DBpedia follows DBpedia
ontology for its classes, thus for type predication, we will use the metric lenient
NDCG@k with a linear decay, adopted from Balog & Neumayer [
Session 3C – SW Challenges: Thursday, 26th October, 2021 13:26 – 13:33 EDT 13:33 – 13:40 EDT 13:50 – 13:55 EDT 13:55 – 14:00 EDT 14:00 – 14:05 EDT 14:05 – 14:10 EDT
Khaoula Benmaarouf, Kanchan Shivashankar, and Nadine Steinmetz
Thang Ta Hoang, Olumide Ebenezer Ojo, Olaronke Oluwayemisi Adebanji, Alexander Gelbukh and Hiram Calvo.
Q & A session for the first two talks
Chaeyoon Kim and Ernesto Jimenez-Ruiz
G P Shrivatsa Bhargav, Dinesh Khandelwal, Saswati Dana and Dinesh Garg
Xiao Ning, Ammar Ammar, Arif Yilmaz Shervin Mehryar, Remzi Celebi
Type Classification over DBpedia and Wikidata Aleksandr Perevalov and Andreas Both
Q & A session for the last four talks
Here again the category prediction will be considered as a multi-class
classification problem and accuracy score will be used as the metric. Wikidata does not
follow a strict ontology for the classes, it has a very large and rather flat set of
classes and subclasses. Thus for type prediction, we use a mean reciprocal rank
(MRR) based scoring system [
Relation prediction is evaluated using the precision, recall and F1 metrics considering the gold standard list relations and predicted list of relations. Table 5 and Table 6 shows the results for the relation prediction task for DBpedia and Wikidata. System Kim et al.
Bhargav et al.
Celebi et al.
Hoang et al.
Steinmetz et al.
Perevalov et al. 0.842 0.825 0.725 0.727 0.734 0.643
In this section, we list the people who organised and contributed to the success of this event.
• Nandana Mihindukulasooriya (IBM Research AI) • Mohnish Dubey (InfAI Dresden) • Alfio Gliozzo (IBM Research AI) • Jens Lehmann (University of Bonn and Fraunhofer IAIS) • Axel-Cyrille Ngonga Ngomo (Universität Paderborn) • Ricardo Usbeck (Fraunhofer IAIS Dresden) • Gaetano Rossiello (IBM Research AI) • Uttam Kumar (University of Bonn) Challenge Programme Committee Members The challenge programme committee helped to peer-review the eight system papers. Each paper received 2 or 3 reviews from the programme committee members and authors took those feedback into account when preparing the camera-ready versions. The organisers would like to thank them for their valuable time.
• Ibrahim Abdelaziz (IBM Research AI) • Carlos Badenes-Olmedo (Ontology Engineering Group, UPM) • Pavan Kapanipathi (IBM Research AI) • Debanjali Biswas (GESIS) • Pablo Calleja (Ontology Engineering Group, UPM) • Jennifer D’Souza (TIB, Leibniz University Hannover) • Uttam Kumar (University of Bonn) • Gaetano Rossiello (IBM Research AI) • Sanju Tiwari (Universidad Autonoma de Tamaulipas) • Ricardo Usbeck (University of Hamburg) • Daniel Vollmers (Paderborn University) • Mohnish Dubey (InfAI Dresden) • Nandana Mihindukulasooriya (IBM Research AI)
We would like to thank the ISWC Semantic Web Challenge chairs, Ernesto Jimenez-Ruiz, Jiaoyan Chen and Despoina Magka, and the whole ISWC organising committee for their invaluable support to make this event a success. We would also like to thank the challenge participants for their interest, quality of work, and informative presentations during the event which made it attractive to the ISWC audience.