SMART 2020 [1] was the first edition of the SeMantic AnsweR Type prediction task (SMART), which part of the ISWC 2020 Semantic Web Challenge. It was co-located with the 19 th International Semantic Web Conference (ISWC 2020) 1 . Given a question in natural language, the task of SMART challenge is, to predict the answer type using a target ontology. The challenge had 2 tracks, one using the DBpedia ontology and the other using Wikidata ontology. Eight teams participated in the DBpedia track and three teams in the Wikidata track. This volume contains peer-reviewed system description papers of all the systems that participated in the challenge. More details about the challenge can be found at https://smart-task.github.io/.
This challenge is focused on answer type prediction, which plays an important role in Question Answering systems. Given a natural language question, the task is to produce a ranked list of answer types of a given target ontology. Previous such answer type classifications in literature are performed as a shorttext classification task using a set of coarse-grained types, for instance, either six types [2,3,4,5] or 50 types [6] with TREC QA task2 . We propose a more granular answer type classification using popular Semantic Web ontologies such as DBpedia and Wikidata.
Table 1 illustrates some examples. The participating systems can be either supervised (training data is provided) or unsupervised. The systems can utilise a wide range of approaches; from rule-based to neural approaches.
Eight teams competed in SMART 2020 and presented their systems at the ISWC 2020 conference. Table 2 shows their presentation titles along with the authors.
Session 6A: Thursday, 5
For each natural language question in the test set, the participating systems are expected to provide two predictions: answer category and answer type. Answer category can be either 'resource', 'literal' or 'boolean'. If the answer category is 'resource', the answer type should be an ontology class (DBpedia or Wikidata, depending on the dataset). The systems could predict a ranked list of classes from the corresponding ontology. If the answer category is 'literal', the answer type can be either 'number', 'date' or 'string'.
Category prediction will be considered as a multi-class classification problem and accuracy score will be used as the metric. As DBpedia follows DBpedia ontology for its classes, thus for type predication, we will use the metric lenient NDCG@k with a linear decay, adopted from Balog & Neumayer [7].
Here again the category prediction will be considered as a multi-class classification problem and accuracy score will be used as the metric. Wikidata does not follow a strict ontology for the classes, it has a very large and rather flat set of classes and subclasses. Thus for type prediction, we use a mean reciprocal rank (MRR) based scoring system [8], where the expected type prediction is a list.
In this section, we list the people who organised and contributed to the success of this event.
We would like to thank the ISWC Semantic Web Challenge chairs, Anna Lisa Gentile and Ruben Verborgh, and the whole ISWC organising committee for their invaluable support to make this event a success. We would also like to thank the challenge participants for their interest, quality of work, and informative presentations during the event which made it attractive to the ISWC audience.
• Nandana Mihindukulasooriya (IBM Research AI)
• Mohnish Dubey (University of Bonn and Fraunhofer IAIS)
• Alfio Gliozzo (IBM Research AI)
• Jens Lehmann (University of Bonn and Fraunhofer IAIS)
• Axel-Cyrille Ngonga Ngomo (Universität Paderborn)
• Ricardo Usbeck (Fraunhofer IAIS Dresden)
The challenge programme committee helped to peer-review the eight system papers and the organisers would like to thank them for their valuable time.