This paper presents our contribution to the SMART challenge 2021 (SeMantic AnsweR Type Prediction Task), speci cally the relation prediction task for both the DBpedia and Wikidata datasets. We introduce our approach to predict the ontology properties (relations) mentioned in a natural language question in order to be able to answer the question correctly. Our solution is based on a pre-trained BERT model using fastai and in combination with data augmentation (for DBpedia). The techniques separately are proven to be very e ective for text classi cation problems and outperform other approaches. In this paper, we used a multi-label classi cation method built-in fastai library for the SMART task, which gives very good results. Achieving high performances for relation prediction is assured by using DBpedia ( 760 classes) and Wikidata class hierarchy ( 50K classes) for results of an experimental evaluation.
This paper describes our approach on relation prediction for natural language
questions within the context of the SMART (Semantic Answer Type and
Relation Prediction Task) challenge 2021 [
Question Answering (QA) systems are commonly used as interface between
a large amount of (un)structured data and users who are enabled to request the
data without knowledge of a formal query language. There are two types of QA:
open and closed domain. Open domain QA systems do not solve speci c topics,
but are used to get the proper answers from various topics in shorter form.
The downside of open domain QA is that it is di cult for the system to get
answers for all possible questions facing various challenges, as e.g. ambiguity, or
incomplete knowledge bases. On the other hand, closed domain QA systems are
focused on particular domains, where the QA application has been developed for
a speci c task, which helps the system to get the answers very fast, and (mostly)
correct. For instance, QA systems in the medical eld (Alzheimer's diseases) [
This paper is structured as following: Section 2 discusses some previous work that is related to our subtask. Section 3 gives an overview of the datasets and Section 4 depicts some results of the analysis of the both datasets (DBpedia and Wikidata). The preprocessing steps and the training steps are discussed in Sections 5 and 6. Evaluation results are described in Section 7. Finally, a conclusion for our results as well as an outlook is described in Section 8. 2
The approach on relation and entity linking by Sakor et. al is based on a set
of rules and a mapping of connected entities and relations [
Abolghasemi et. al proposed an instance-based method to detect the relation
of a new question using similar paraphrases of questions in the training data
[
Zhao et. al proposed a solution to solve the problem of incompleteness of
KGQA, as the researches are focused on processing each problem independently,
without taking the hidden relations inherit from the neighborhood in their
consideration [
The problem of QA has been investigated by Mohammed et. al, where the
authors aimed to focus on accuracy-complexity tradeo , as simple
straightforward baselines CNN and GRUs were used plus a few heuristics on the
SimpleQuestions dataset [
Since its publication, BERT [
The SMART task provides datasets for the two KBs DBpedia and Wikidata. Some statistical details on the datasets provided for the challenge are shown in Table 1 and Section 4. Our approach considers the task as a relation prediction classi cation, where each question is assigned a relation category. While this task is considered a short-text classi cation, what makes the classi cation challenging is a few unique characteristics of the datasets which contribute to data sparsity. For the challange, for both ontology tasks the following datasets are provided: relation vocabularies, train data, and test questions. To train a model on the data, it needs to be transformed into a feature-target form. 3.1
The DBpedia dataset consists of 42,756 samples, which is split into 80% as training data, and 20% as testing as shown in Table 1. The dataset is divided into three les (relation vocabulary, test questions, and train data). The relation vocabulary contains properties from the mapped ontology1 and unmapped properties2. Moreover, the total number of relations in the vocabulary is 717. The train le contains the following four attributes: question, relations, number of relations and ID. The number of relations speci es how many di erent classes of relations are contained in the questions - which results in one list of relevant relations per class. The test questions le only contains the questions and the ID. A sample of the DBpedia train data is shown in Fig ??. Wikidata consist of 24,112 as training data, and 6,029 as testing data as shown in Table 1. The train data contains ve attributes: questions, relations, relation labels, num of rels, id. For Wikidata, ontology properties have a unique identi er3 which is not human-readable (attribute relations) and additionally human-readable labels (attribute relation labels). The length of the vocabulary is 3,639. Figure 2 shows ve sample records from the training dataset.
We analyzed the training datasets for the frequencies of the occurring properties to be able to assess the distribution of properties and the sparsity of data for 1 having http://dbpedia.org/ontology/ as pre x 2 having http://dbpedia.org/property/ as pre x 3 usually starting with a P, such as P2397 for the property with the human-readable label \YouTube channel ID" speci c classes (class as in classi cation of relation sets). As shown in Figure 3 the distributions of properties is long-tailed. The most frequent properties are dbo:genre and dbp:birthPlace respectively. Out of 338 unique mapped properties, 42 only occur once. For the unmapped properties these numbers are 357 and 58 respectively.
(a) (b) Figure 4 shows the most frequently used labels in the training dataset, where instance of is the most frequent relation with 6,418 occurrences which is 4 times higher than the second most frequent relation (point of time with 1,314 occurrences). Out of 3,171 unique relations in the dataset, a large amount of 1,889 only occurs once. 4.3
As shown in the previous sections, the training datasets are very imbalanced regarding the distribution and frequency of relations throughout the dataset. This results in low accuracies for questions referring to relations in the long tail of the distribution. Although not considered for the approach presented in this paper, we examined strategies to compensate such imbalances. Unfortunately, simple data augmentation strategies, as discussed in Section 5.2 do not su ce. The training dataset requires to be enriched with questions that contain the long tail relations, but with di erent contexts and wording than contained in the dataset. We consider this future work to further improve the results of our approach. 5
Our analytical approach includes several processing steps. For the di erent classi cation processes for DBpedia and Wikidata, we utilize the same preprocessing pipeline except in the rst step. For DBpedia, we remove the pre x from the relation labels, then lower casing the letters, and resolve the camel case format of the labels. For example, dbo:RecordLabel is transformed to record label. After that, the pipelines for Wikidata and DBpedia are the same. In the next step, the pipeline takes the questions, relations, and blocks throw data parsing, data augmentation, remove duplicates inside the block, create an indexation between the index of the label inside vocabulary and their block index. Labels that exist in the vocabulary and do not exist in the training set are added. A binary matrix is created and then the labels are merged inside the sentence randomly. The training dataset is split into train and validation sets and nally, the text is tokenized using the BERT tokenizer. 5.1
Questions, relations, and block parsing is done on training data in the way that the questions syntactic order is shu ed in three di erent orders and then combined all the three di erent types of questions. Indexation is used between vocabulary and their related blocks to get the index values to speed up the process. We utlize lists consisting of: question, length, block, relations.Training data Q is appended into these lists separately and then the question list is split into Q1 and Q2. The subsequent data is appended randomly into splited question lists. We remove duplicates that cause redundancy, as this will lead to easier computations for the model to nd patterns from unique blocks, without being biased to one block instead of the other because it is redundant. The matrix is transformed to binary format to be tted for the training. 5.2
To extend the training data, several augmentation strategies can be applied
[
For the tokenization of the input data, we utilized the FastAIBertTokenizer from
the fastai library [
Most of the modern NLP systems utilized gated recurrent neural networks
(RNNs), such as long short-term memory (LSTMs) and Gated Recurrent Units
(GRUs), with additional attention mechanisms before the release of
transformers [
In 2017, an encoder-decoder model called transformer was introduced to solve
the problems that facing RNNs. Transformers can be used in classi cation
problems that are considered supervised learning, as in our case. For our approach
the BERT model is used which is a model from the transformers family. We
utilized BERT BASE UNCASED which is not case-sensitive and has a much lower
number of layers compared to the BERT LARGE model, as base models have only
12 layers in Encode, and Decoder, with a total number of parameters 110M [
The loss function algorithm is used as binary cross entropy with logistic losses which applies a sigmoid activation layer to the output of the binary cross entropy layer to be mapped to 0 or 1. The binary cross entropy is preferred over multilabel entropy because of higher accuracies.
The evaluation metric used to evaluate the model is F1-score. The simple accuracy metrics cannot be used, as it is not taking into consideration the imbalance of the dataset, while F1-score uses precision, and recall getting a score out of 100% to know how good the model could predict each label. The learning rates are set to 6 di erent rates.
The maximum sequence length was set to 256, with a batch size of 32, and the model is trained for 20 iterations.
In terms of training and validation, we utilized two di erent validation methods: static 80/20 split and k fold cross validation with 3 folds. As shown in our results in the next section, we achieved better results with the cross validation method for the Wikidata dataset, but not for DBpedia. 7
The results from di erent combinations of strategies are shown in Table 2. We utilized the basic approach with trained model and in addition data augmentation for the increase of training data and three di erent validation split methods: 80/20 split, 99/1 split and cross validation as a exible version for the validation step. Obviously, the quality of the results did not increase for both datasets using all additional strategies. The cross validation strategy did not achieve better results for DBpedia, but for Wikidata. For DBpedia, the 80/20 split achieved better results than 99/1 split. Whereas the data augmentation step was only successful for DBpedia. For Wikidata, the recall decreases signi cantly using the augmented training dataset. While we were not able to identify the exact reason for that behavior, we noticed a signi cantly increased amount of predicted relations when utilizing the augmented training dataset { an average of 9 relations are predicted compared to the results with the best F1 score having only 2 relations predicted at average.
The pipelines of the best achieving combinations are depicted in Figure 5. Overall, our best strategy combinations could outperform the other competitor of the SMART 2021 challenge, as shown in Table 2. 8
In this paper, we presented our approach for the SMART Task challenge of ISWC 2021 for the Relation Prediction Task. The goal was to predict a set of relations relevant to create the formal SPARQL query to be able to answer the question. We created a classi cation pipeline and additionally implemented data augmentation and cross-validation methods. The results of our experiments show di erent combinations of the strategies for both datasets { Wikidata and DBpedia. The combination of di erent strategies achieved very good results compared to the other participant of the relation prediction task. We consider the problem as a set of sequence classi cation tasks, each one making use of a ne-tuned BERT classi er. For the more ne-grained (and more challenging) problem of Relation Prediction (since the classes can be hundreds or thousands), we have proposed the enrichment of the BERT trained model with additional strategies. For future work, we consider a more adaptive strategy to deal with the imbalanced datasets and utilize data augmentation only for the long tail of the properties in terms of the frequency distribution. Also, for Wikidata, the relation labels should be considered instead of the IDs - although they are the ones required for the classi cation task. 9
This work was partially funded by the German Research Foundation (DFG) under grant no. SA 782/30-1 and STE 3033/1-1.