SMART 2020 [1] was the first edition of the SeMantic AnsweR Type prediction task (SMART), which part of the ISWC 2020 Semantic Web Challenge. It was co-located with the 19th International Semantic Web Conference (ISWC 2020)1. Given a question in natural language, the task of SMART challenge is, to predict the answer type using a target ontology. The challenge had 2 tracks, one using the DBpedia ontology and the other using Wikidata ontology. Eight teams participated in the DBpedia track and three teams in the Wikidata track. This volume contains peer-reviewed system description papers of all the systems that participated in the challenge. More details about the challenge can be found at https://smart-task.github.io/. This challenge is focused on answer type prediction, which plays an important role in Question Answering systems. Given a natural language question, the task is to produce a ranked list of answer types of a given target ontology. Previous such answer type classifications in literature are performed as a shorttext classification task using a set of coarse-grained types, for instance, either six types [2, 3, 4, 5] or 50 types [6] with TREC QA task2. We propose a more granular answer type classification using popular Semantic Web ontologies such as DBpedia and Wikidata. Table 1 illustrates some examples. The participating systems can be either supervised (training data is provided) or unsupervised. The systems can utilise a wide range of approaches; from rule-based to neural approaches.
movies directed by and starring William Shatner.
Which programming languages were influenced by Perl? Who is the heaviest player of the Chicago Bulls? How many employees does Google have?
dbo:Actor
Wikidata wd:Q33999 dbo:ProgrammingLanguage wd:Q9143 dbo:BasketballPlayer xsd:integer wd:Q3665646 xsd:integer 1https://iswc2020.semanticweb.org/ 2https://trec.nist.gov/data/qamain.html
Eight teams competed in SMART 2020 and presented their systems at the ISWC 2020 conference. Table 2 shows their presentation titles along with the authors.
Slot
For each natural language question in the test set, the participating systems are expected to provide two predictions: answer category and answer type. Answer category can be either ‘resource’, ‘literal’ or ‘boolean’. If the answer category is ‘resource’, the answer type should be an ontology class (DBpedia or Wikidata, depending on the dataset). The systems could predict a ranked list of classes from the corresponding ontology. If the answer category is ‘literal‘, the answer type can be either ‘number’, ‘date’ or ‘string’.
DBpedia Dataset
Category prediction will be considered as a multi-class classification problem
and accuracy score will be used as the metric. As DBpedia follows DBpedia
ontology for its classes, thus for type predication, we will use the metric lenient
NDCG@k with a linear decay, adopted from Balog & Neumayer [
Setty et al
Nikas et al
Perevalov et al
Kertkeidkachorn et al
Ammar et al
Vallurupalli et al
Steinmetz et al
Bill et al
Here again the category prediction will be considered as a multi-class
classification problem and accuracy score will be used as the metric. Wikidata does not
follow a strict ontology for the classes, it has a very large and rather flat set of
classes and subclasses. Thus for type prediction, we use a mean reciprocal rank
(MRR) based scoring system [
MRR 0.68 0.59 0.40
In this section, we list the people who organised and contributed to the success of this event. Challenge Chairs • Nandana Mihindukulasooriya (IBM Research AI) • Mohnish Dubey (University of Bonn and Fraunhofer IAIS) • Alfio Gliozzo (IBM Research AI) • Jens Lehmann (University of Bonn and Fraunhofer IAIS) • Axel-Cyrille Ngonga Ngomo (Universität Paderborn) • Ricardo Usbeck (Fraunhofer IAIS Dresden) Challenge Programme Committee Members The challenge programme committee helped to peer-review the eight system papers and the organisers would like to thank them for their valuable time. • Ibrahim Abdelaziz (IBM Research AI) • Sarthak Dash (IBM Research AI) • Srinivas Ravishankar (IBM Research AI) • Pavan Kapanipathi (IBM Research AI) • Md Rashad Al Hasan Rony (Fraunhofer IAIS) • Liubov Kovriguina (Fraunhofer IAIS) • Mohnish Dubey (University of Bonn and Fraunhofer IAIS) • Nandana Mihindukulasooriya (IBM Research AI)
We would like to thank the ISWC Semantic Web Challenge chairs, Anna Lisa Gentile and Ruben Verborgh, and the whole ISWC organising committee for their invaluable support to make this event a success. We would also like to thank the challenge participants for their interest, quality of work, and informative presentations during the event which made it attractive to the ISWC audience.