In this paper we describe our submission to the SMART 2021 Answer Type Prediction task. We propose a BERT based solution to the problem. The proposed approach relies on type embeddings obtained based on the type names. It allows our model to predict types at test time that were not seen during training. Analysis of the training dataset reveals the presence of noise in the labels. Therefore, we develop a label augmentation scheme to reduce the noise in the annotations and increase the quality of the training data. Our model trained on the de-noised data achieves 0.986 accuracy on the answer category prediction task and 0.825 and 0.790 NDCG@5 and NDCG@10 respectively on the test sets.
In this paper, we focus on the SMART2021-AT DBpedia dataset. The dataset has 40621 samples for training and validation. An additional 10093 samples are held out for testing. We perform elementary data cleanup (for example, removing samples with null labels, removing duplicates, etc) and split the dataset into training and validation sets in the ratio 80 : 20. Table 2 summarizes the size of di erent sets.
To establish a performance (accuracy and NDCG@k) upper-bound on this dataset, we performed an experiment where we evaluated1 the training and validation sets against themselves. The goal was to check what accuracy and NDCG@k a system would obtain if its predictions exactly matched the gold annotations. On the training set, we found that the category prediction accuracy was 1 whereas, NDCG@5=0.8261 and NDCG@10=0.7529. This indicates that the gold label set is not complete. Such incompleteness/noise in the training data will directly impact the model's performance. Upon inspection, we found that some of the ancestor types (also known as super types, parent types) were missing in the gold types list. In some training samples, the types were not sorted according to the descending order of their depths in the ontology. We modi ed the gold type list of each training sample in the following ways - (i) We completed the type list by adding the missing ancestor types. (ii) We sorted the completed type list in descending order of their depth. We refer to these two steps collectively as label augmentation. Table 3 summarizes the impact of each of the above steps. The metrics in Table 3 also serve as a soft upper-bound on the performance of any model trained on this dataset. We train our models on the modi ed training set (+ ancestor types + sorting) but we validate on the unmodi ed validation set. 1 github.com/smart-task/smart-dataset/blob/master/evaluation/dbpedia/evaluate.py The metric NDCG@k is sensitive to the ordering of the predicted types. The evaluation script expects the predicted types to be sorted from the nest to the coarsest (i.e, decreasing order of depth in the ontology). But in several samples, there are multiple types of the same depth. In such cases, their ordering in the predicted type list can be arbitrary. This phenomenon could be the reason why the train and validation NDCG@k are not 1. 3
In this section we will describe our proposed approach to solve the Answer Type Prediction Task. 3.1
To simplify the modelling task, we work on an equivalent reformulation of the problem. The reformulated problem can be stated as follows - Given a natural language question, the rst task is to predict the answer category from the set of labels C = fresource; number; date; string; booleang. If the predicted category is resource, then the second task is to rank the set of DBpedia types T = ft1; t2; : : :g from most relevant to least relevant. We train the model on the reformulated task, transform its predictions and report the metrics on the original task. 3.2
We embed the given natural language question Qi =< qi1; qi2; : : : > (where qij are the tokens) into vector space and use this embedding to predict the categories and rank the types. We leverage BERT to obtain the question embeddings as follows: (i) We surround the question tokens with special tokens [CLS], [C] and [SEP] to obtain a sequence of the form <[CLS][C]qi1; qi2; : : :[SEP]> (ii) The above sequence is passed through BERT to obtain a sequence of vectors < v[CLS]; v[C]; vqi1 ; vqi2 ; : : : >.
v[CLS] is used for the purpose of ranking the set of types T whereas v[C] is used to predict the answer category from the set of labels C. The vector v[C] is passed through a fully connected layer (with parameters W and b) followed by a softmax layer in order to obtain the probability for each of the categories in C. To tune the parameters for this task, we use the cross entropy loss.
LC =
X
log p(ci jQi)
where Dtrain is the training set and ci is the true category for the question Qi. 3.4
In order to rank the types T in the next step, we require a vector embedding ei for every type ti 2 T . We use BERT to obtain the initial embeddings of the types. To do so, we rst normalize the names of each type ti to obtain English phrases. For example, the type \dbo:GovernmentalAdministrativeRegion" is transformed to \Governmental Administrative Region". The normalized type names are passed through BERT and the resulting output corresponding to the [CLS] vector is used as the initial type embedding. Thus, we obtain an embedding matrix ET 2 RjT j d, where d is the embedding dimension. Each row ei of ET corresponds to the embedding of the type ti. By creating the type embeddings this way, we ensure that we will have a good representation of all the types even though they may be unseen during training. 3.5
For a given question Qi, we predict the probability p(tj = 1jQi) for each type tj and then rank the types from the most probable to the least. p(tj = 1jQi) is computed as follows: p(tj = 1jQi) =
1 1 + exp( v[TCLS] etj )
We train the model to maximize the probabilities of the gold types using the following loss:
LT =
X
1res(Qi)
X tj2T 11Qi (tj ) log p(tj jQi) where, Dtrain is the training dataset, 1res(Qi) indicates whether the category of question Qi is \resource" or not, 1Qi (tj ) indicates whether the type tj is a valid answer type for the question Qi, 1 and 2 are scalar hyperparameters. In every training sample, the number of negative types is far greater than the number of positive types. Due to this imbalance, the model will learn to predict zero probability for all the types. To remedy this, we use the hyperparameters 1 and 2 to balance the losses corresponding to the positive and negative types.
We jointly optimize LC and LT in a multi task fashion. The multi-task learning objective function is:
L = LC +
LT where is a scalar hyperparameter that controls the relative importance of LC and LT . The parameters of the question encoder BERT, W , b and ET are all updated during training. 3.7
At the inference time, we rst run the answer category prediction. The type
prediction depends on the predicted category. Table 4 illustrates how the output
on the reformulated task is converted to the output on the original task.
We implemented the proposed approach in Pytorch. The source code and the
trained models have been released on Github2. BERT-Base is used in all our
experiments. We use ADAM [
BERT 0.986 0.810 0.771
BERT + label augmentation 0.986 0.820 0.785 0.985 0.825 0.790 4.3
We performed analysis on the validation set to understand the strengths and weaknesses of our model (BERT + label augmentation).
Table 7 shows the confusion matrix on the answer category prediction task. Table 8 shows randomly sampled examples of each kind of mistake made by the classi er. In all cases where the model predicts boolean instead of literal and boolean instead of resource, the model is correct and the gold annotation in incorrect. The confusion between resource and literal is prominent but hard to resolve. The answer category (resource or literal) is completely dependent on the knowledge base. It is impossible to decide resource vs literal without nding the answer to the question rst.
To study the errors made by the answer type predictor, we randomly sample the validation set examples on which the model's NDCG@5 is less than 0.2. We show some of these examples in table 9. In examples 1 and 3, we see that the model's ranking is correct but the gold annotations are incorrect. Example 2 however, is a mistake by our model and the reason is unclear. resource
Questions How many seats are in prefectural assembly? What is the demised place of Leo III What is the symbol for pi? what year did tim duncan enter the nba Did Barbados have a diplomatic relationship with Nigeria in the past? Was Natalia Molchanova born in the Bashkir Autonomous Soviet Socialist Republic? is ANZUS a signatory? Was Gustav Mahler`s birth place located in the administrative territorial entity of Kalista ? In what country is Mikhail Fridman a citizen? What's the original language for Close Encounters of the Third Kind? Who sponsors the FC Bayern Munich? Did Masaccio die before the statement of Gregorian Is the language of Neptune, Czech? Is Thom Enriquez part of the lm crew for Beauty and the Beast? Is there an audio recording of Charles Duke?
What is the geography of the planet, Mars? In this paper we explored how BERT can be used to address the problem of answer type prediction. We rst established a soft upper bound on the performance of models that are trained on the SMART 2021 dataset. We developed a label augmentation scheme to de-noise the gold annotations and hence improve the model. Our model achieves 0.986 accuracy on the answer category prediction task and 0.825 and 0.790 NDCG@5 and NDCG@10 respectively on the test set. On the validation set, the performance of our model (after label augmentation) is only 0.068 and 0.019 short of the soft upper-bound established in Table 3. We also present error analysis that shows the nature of errors made by our model and the noise in the dataset.