Answer type prediction is a key task in Question Answering (QA) that aims at predicting the type of the expected answer for a user query expressed in natural language. In this paper we focus on semantic answer type prediction where the candidate types come from a class hierarchy of a general-purpose ontology. We model the problem as a two-stage pipeline of sequence classi cation tasks (answer category prediction, answer literal/resource type prediction ), each one making use of a ne-tuned BERT classi er. To cope with the harder problem of answer resource type prediction, we enrich the BERT classi er with a rewarding mechanism that favors the more speci c ontology classes that are low in the class hierarchy. The results of an experimental evaluation using the DBpedia class hierarchy ( 760 classes) demonstrate a superior performance of answer category prediction ( 96% accuracy) and literal type prediction ( 99% accuracy), and a satisfactory performance of resource type prediction ( 78% lenient NDCG@5).
Question Answering (QA) is a task in the eld of Natural Language Processing
and Information Retrieval that aims at automatically answering a question posed
by a human in a natural language [
In this paper, we focus on a two-stage answer type prediction task where a rst step aims at nding the general category of the answer (resource, literal, boolean), while a second step tries to predict the particular literal answer type (number, date, or string, if the predicted category of the rst step is literal ), or the particular resource class (if the predicted category of the rst step is resource). We consider the case where the resource classes belong to a rich class hierarchy of an ontology containing a large number of classes (e.g., >500), and model the Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). problem as a set of sequence classi cation tasks, each one making use of a netuned BERT model. For the more ne-grained (and thus more challenging) task of resource class prediction, we propose to enrich the BERT classi er with a rewarding mechanism that favors the more speci c ontology classes that are low in the class hierarchy. Fig. 1 depicts this two-stage answer prediction task, the classi ers we use in each di erent sub-task, and the accuracy of the obtained results. The evaluation results using the DBpedia class hierarchy ( 760 classes) and a ground truth of 40,393 train questions for category prediction, 17,571 for resource/literal type prediction, and 4,393 test questions demonstrate the high performance of our approach. Speci cally, we achieve 96.2% accuracy on answer category prediction, 99.2% accuracy on literal type prediction, and 77.7% NDCG@5 on resource type ranking.
The rest of the paper is organized as follows: x2 describes the context, x3 describes our approach, x4 reports the results of the evaluation, and nally, x5 concludes the paper. 2
The context of this work is the SMART (SeMantic AnsweR Type) challenge of
ISWC 20201 [
Two datasets are provided for this task, one using the DBpedia ontology and the other using the Wikidata ontology. Both follow the below structure: Each 1 https://iswc2020.semanticweb.org/program/semantic-web-challenges/ question has a (a) question id, (b) question text in natural language, (c) an answer category (resource/literal /boolean), and (d) answer type. If the category is resource, answer types are ontology classes from either the DBpedia ontology ( 760 classes) or the Wikidata ontology ( 50K classes). If the category is literal, answer types are either number, date, or string. Finally, if the category is boolean, answer type is always boolean.
An excerpt from this dataset is shown below: [ { "id": "dbpedia_14427", "question": "What is the name of the opera based on Twelfth Night?", "category": "resource", "type": ["dbo:Opera", "dbo:MusicalWork", "dbo:Work" ] },{ "id": "dbpedia_23480", "question": "Do Prince Harry and Prince William have the same parents?", "category": "boolean", "type": ["boolean"] } ]
With respect to the size of the datasets, the DBpedia dataset contains 21,964 questions (train: 17,571, test: 4,393) and the Wikidata dataset contains 22,822 questions (train: 18,251, test: 4,571). The DBpedia training set consists of 9,584 resource, 2,799 boolean, and 5,188 literal questions. The Wikidata training set consists of 11,683 resource, 2,139 boolean, and 4,429 literal questions. 3
Here we describe our approach for answer type prediction: in x3.1 we provide some background, in x3.2 we describe question category prediction, in x3.3 we describe literal answer type prediction, and in x3.4 we describe resource answer type prediction. The models and code are publicly available at: https://github. com/cnikas/isl-smart-task. 3.1
BERT [
A question can belong to one of the following three categories: (1) boolean, (2) literal, (3) resource. Boolean questions (also referred to as Con rmation questions) only have `yes' or `no' as an answer (e.g. \Does the Owyhee river ow into Oregon?"). Thus, there is no further classi cation for this category of questions. Resource questions have a speci c fact as an answer (e.g. \What is the highest mountain in Italy?") that can be described by a class in an ontology (e.g. http://dbpedia.org/ontology/Mountain). Literal questions have a literal value as answer, which can be a number, string, or date (e.g. \Which is the cruise speed of the airbus A340?").
To detect question categories, we ne-tune a BERT model using the Huggingface PyTorch implementation2. We choose this model because we approach answer type prediction as a classi cation problem where each question is a sequence of words. To ne tune BERT we used the training datasets provided for the SMART challenge (described in x2). Speci cally, we used questions from both the DBpedia and the Wikidata dataset. Because the data is imbalanced for categories (13.7% boolean, 26.6% literal, 59.4% resource) we randomly sampled questions for each class so that all classes had the same number of samples.
As we will see below, this model achieves 96.2% accuracy on our test set in this prediction task. 3.3
The answer type for questions that belong in the literal category can be: 1) a date, i.e. a literal value that describes a date, 2) number, i.e. a numeric value, or 3) a string, i.e. a text value. Due to the small number of classes (3), it is very e ective to train a language model. We again use a ned-tuned BERT model to classify literal questions in one of the 3 types. Similar to question category prediction, we used questions from both the DBpedia and the Wikidata dataset and also randomly sampled questions for each class to cope with class imbalance (29.1% date, 27.3% number, 43.6% string). As we will see, the model achieves 99.2% accuracy for literal questions in our test set. 3.4
The prediction of the answer type of questions in the resource category is a more ne-grained (and thus more challenging) classi cation problem, because of the large number of types a question can be classi ed to ( 760 classes on DBpedia and 50K classes on Wikidata). Therefore, it is not e ective to train a classi er on all the ontology classes, especially for open-domain tasks.
To reduce the number of possible types for classi cation, we selected a subset (C) of all ontology classes, based on the number of samples of each class in the training set. This subset C contains classes that have at least k occurrences in 2 https://huggingface.co/transformers/ the training set. We set k = 10 as this number provides a good trade-o between number of classes and performance.3 The choice of this parameter is described more extensively in section 4.2. The nal number of classes in C is 88. Because we chose to train the system on a subset of all the classes, our classi er cannot handle questions with labels that are not included in this subset. To tackle this problem, we replace their labels with the labels of super classes that belong in C. Then we ne tuned a BERT model on them.
Since most questions in the dataset have several answer types ordered by
speci city, according to the semantic hierarchy formed in the ontology, in the
ne tuning stage we use these questions multiple times, one with each of the
provided types as the label. The goal is to nd an answer type that is as speci c
as possible for the question. However, the model may classify a question to a
more general answer type in the ontology. To tackle this problem, we `reward'
(inspired by [
We report results for the following metrics:
{ Accuracy, for category prediction (the percentage of questions classi ed in
the correct category).
{ Precision, for type prediction (the percentage of the questions for which the
top type found by the system was one of the types provided in the test
dataset, without considering type speci city).
{ Lenient NDCG@k (with a Linear decay) [
Lenient NDCG@k, which has been introduced in [
DCG . Finally we compute and report the and normalized DCG (nDCG) as iDCG average nDCG over all questions in the test dataset. 4.2
Initially, we had no access to the nal test dataset of the SMART challenge, so we used 90% of the DBpedia training set4 as our training dataset and the remaining 10% as our test dataset. For category prediction and literal type prediction we also use the questions from the training dataset for Wikidata for training the classi ers. Our approach achieved the results shown in Table 1. We notice a superior performance of category prediction (96.4% accuracy) and a very high performance of type prediction (83% precision and 79% lenient NDCG@5).
Running the same experiments without the rewarding mechanism, we notice an around 2% drop in the performance (Lenient NDCG) of literal/resource type prediction. Accuracy (category prediction) Precision (literal/resource/boolean type prediction) Lenient NDCG@5 with linear decay (literal/resource type prediction) Lenient NDCG@10 with linear decay (literal/resource type prediction) Tuning of the k parameter To nd the optimal value for the parameter k, which is the minimum sample size required to include a class in the subset of classes included in the classi er, we evaluated our system using 4 di erent values: 5, 10, 30 and 50. Table 2 shows the number of classes included in the classi er for each di erent value of k and the corresponding performance. We notice that the best results are obtained using k=10, while the results for all other cases are slightly worse.
Error analysis. To better understand the classi cation performance of category prediction, literal type prediction, and resource type prediction, we inspected their confusion matrices. The results are shown in Table 3. As regards 4 https://github.com/smart-task/smart-dataset/tree/master/datasets/ DBpedia category prediction, we see that our system classi es in the correct category 99% of the boolean questions, 92% of the literal questions, and 98% of the resource questions. For literal type classi cation, our system classi es in the correct type 98.4% of date questions, 99.5% of number questions, and 99.5% of string questions. We notice that, for category prediction, most errors occur between the classes literal and resource. For instance, 41 questions of literal type are misclassi ed as of type resource. As regards resource type prediction, the table shows the confusion matrix for the top-5 (most frequent) resource classes. We notice that there is signi cant confusion between the classes City and Country, as well as between the class Person and other classes.
By manually inspecting several of the misclassi cation cases, we noticed that some of these errors occur on questions where the correct category is very ambiguous, such as the question \In what area is Fernandel buried at the Passy Cemetery?" (labeled as a literal question with type `string', while our system classi es it as a resource question of type `dbo:Place'), or the type provided in the dataset is wrong, e.g. the question \What did the pupil of Mencius die of ?" is labeled as a literal question with type `date', while our system predicts that the question category is resource and `dbo:Disease' is one of the predicted classes. 4.3
After the nal test dataset was released, we evaluated our system again, using the script provided by the challenge organizers. We obtain the results shown in Fine-tuning. We ne-tuned the models on Google Colab5, a Jupyter notebook environment that runs in the cloud and o ers access to GPUs. With a batch size of 32, number of epochs set to 3 and using an Nvidia Tesla K80 GPU, the time required for ne-tuning each classi er is: 49 mins and 25 secs for the resource question type classi er, using 26,259 questions, 27 mins and 51 secs for the question category classi er, using 14,814 questions, and 15 mins and 3 secs for the literal question type classi er, using 8,025 questions.
Execution. To classify a question into a category and predict its answer type, we execute the system locally on a machine with 2 cores and 8 GB of RAM, without using a GPU. While the system is running, it requires approximately 2.3 GB of RAM to load the 3 classi ers in memory.
This means that the proposed approach has low main memory requirements.
Moreover, this memory footprint can be further reduced if we use a smaller
and lighter language model, such as DistilBERT [
We plan to integrate the proposed classi cation models in the Question
Answering module of Elas4RDF [
Screenshots of the system for the query \Greek philosopher from Athens who is credited as one of the founders of Western philosophy" are shown in Figure 2.
Moreover the classi cation model presented in this paper can be exploited also in the \Schema perspective", that shows the classes of the top-ranked triples (for allowing the user to re ne as she wishes to), in order to promote (or just mark) the class that corresponds to the predicted answer type. 5 https://colab.research.google.com/
Fig. 2. Application Context: Elas4RDF
A demo of Elas4RDF over DBpedia [
We have presented an approach for semantic answer type prediction, an important sub-task of QA which splits the problem into a two-stage pipeline of classi cation tasks: answer category prediction and answer literal/resource type prediction. We model the problem as a set of sequence classi cation tasks, each one making use of a ne-tuned BERT classi er. For the more ne-grained (and more challenging) problem of answer resource type prediction (since the classes can be hundreds or thousands), we have proposed the enrichment of the BERT model with a rewarding mechanism that considers the hierarchy of the ontology classes, favoring the more speci c classes that are low in the class hierarchy. The evaluation results demonstrated the performance of the proposed method, achieving >96% accuracy in predicting the general answer category, >98% accuracy in predicting the literal type, and >77% NCDG@5 in ranking the predicted resource classes.
Our results showcase that it is feasible to achieve ne grained answer type prediction with very high precision and without expensive computations.
Issues that are worth further research include: methods for ne-tuning the parameter k that determines the minimum amount of training data needed to obtain a certain degree of performance, and evaluating the rewarding scheme in di erent datasets, e.g. in knowledge bases that have ontologies with more deep class hierarchies.