In this paper, we address the Task 1 (of the SMART Task 2021) of predicting the answer categories and types based on target ontologies, which could be useful in knowledge-based Question Answering (QA) systems. We introduced our method by combining the power of BERT architectures with data oversampling via replacements of linked terms to Wikidata and dependent noun phrases to attain the state-ofthe-art performance. The accuracy on the DBpedia dataset is 98.5%, whereas NDCG@5 and NDCG@10 are 72.7% and 66.4% respectively. Our model has the best performance compared to other teams, with the accuracy score of 98% and Mean Reciprocal Rank (MRR) of 70% on the Wikidata dataset.
In Natural Language Processing (NLP), Knowledge Base Question Answering
(KBQA) is a task that involves searching for correct answers to a given natural
language question using knowledge bases (KB). While this task appears easy
to humans, it becomes a challenge for machines to detect the semantics in the
questions and match them to the KBs before choosing which answer is the best.
Language diversity, as well as semantic meanings, are barriers that create
lexical gaps between questions and answers in reality. At the moment, there are
two main approaches: semantic parsing-based (SP-based) methods and
information retrieval-based (IR-based) methods, with deep neural networks making
substantial contributions to improving their performance. [
In SMART 2021 Semantic Web Challenge3, we engage Task 1 { category
and type prediction over two datasets, Wikidata and DBpedia [
natural language question, our task is to build models that can predict answer
category and answer type by relying on a set of candidates from a target ontology.
There are 3 answer categories, "resource", "literal" and "boolean". If the
answer type is "boolean", the answer category will also be "boolean". If the
answer category is "literal", the answer type can be either "number", "date"
or "string". If the answer category is "resource", the answer type will be an
ontology class (DBpedia or Wikidata). Category prediction will be evaluated by
accuracy scores across all datasets. A metric lenient NDCG@k [
In this paper, we apply BERT classi ers [
As the task description of this challenge, each question should belong to a unique
category class. As a result, category prediction is a multiclass classi cation
problem. On the other hand, type prediction is a multilabel classi cation in which a
given question can have multiple answer types. With answer categories and types
predicted, a KBQA system can reduce the searching time for possible answers
in the data [
If we consider category and type prediction as a problem of text classi
cation, there is a wide range of methods from logistic regression (LG), Naive
Bayes or Bayes networks [
Referring to some of the papers from the previous years' challenge [
KBQA corpora are usually imbalanced with numerous natural language
questions, including rare questions created from language diversity and human
creativity. Therefore, oversampling and undersampling techniques are helpful to
reduce popular data and increase rare data until the dataset distribution is
more balanced or less biased. There are many methods known for oversampling
techniques such as SMOTE [
To determine the proper approach, we will rst do a preliminary analysis of the datasets. Both Wikidata and DBpedia datasets are unbalanced and are classi ed into three categories: resource, literal and boolean. The resource category accounts for the majority of datasets, followed by literal boolean categories. We conduct some investigations on older datasets, which are not signi cantly di erent from newer datasets with noise removed.
In the DBpedia dataset, we found a lot of questions with "empty" or None
answer types, as shown in Figure 1. One of the solutions for this problem is
that we can use a natural language inference (NLI) approach [
Type prediction is not di cult with the boolean and literal categories because their answer types are simple. There are only a single type (boolean) for the boolean category and three types: (date, number, and string) for the literal category. In contrast, it is problematic with the resource category because it contains 376 types of DBpedia and 3308 types of Wikidata. To reduce the number of types in Wikidata, we only take the rst type of answer that will be used for the experiment. We can also use ontology taxonomy of Wikidata to lter the redundant types (parent-child cases) to get the lesser number of types, but we will not do so in this work.
The imbalanced features in the datasets appear in the answer types as shown in Figure 2. When taking the rst type, which is prepared for the training process, we notice that there are still a lot of types with only a few questions. There are 2032 types and 128 types corresponding to Wikidata and DBpedia datasets, respectively, if counting response types with less than 5 questions. Clearly, it is a challenge for text classi cation methods to detect enough di erences between rare types. We debated whether to remove these rare types or keep them because they do not contribute so much to the classi cation performance. Finally, we decided to keep these types and apply an oversampling technique to increase the number of questions of each type as many as possible, with the expectation that the datasets are less imbalanced than before.
For each question, spaCy4 v.2.3.2 was used to analyze the question structure to get its semantic units, such as question type, subject, main verb (also ROOT), terms (noun phrases, dependent noun phrases) to build the sentence template, and apply entity linking (EL) methods to extract terms connected to Wikidata. For spaCy components, we take en core web lg, STOP WORDS, lemmatizer (Lemmatizer, ADJ, NOUN, VERB), and sentencizer pipeline.
The sentence template was built by a greedy algorithm which absorbs all longest terms. First, we rank the terms according to their length. Then, for each term, we search it in the sentence. If found, we put a pair of opening and closing curly brackets f...g around it to form the sentence template. This algorithm will ignore terms that are contained within larger ones.
We realized that the question structure is also useful for Task 2 of this challenge, which involves predicting the relations of questions. To easier understand what we analyzed, take a look at the following example. { "question": "What periodical literature does Delta Air Lines use ,! as a moutpiece?", "category": "resource", "type": ["publication", "recurring", "intellectual work", "text", ,! "communication medium", "serial"], "question_template": "What {periodical literature} does {Delta Air ,! Lines} {use} as {a mouthpiece}?", "key_terms": ["periodical literature", "Delta Air Lines", "use", ,! "a mouthpiece"], "subject": ["Delta Air Lines"], "main verb": ["use"], "entities": ["literature", "a mouthpiece", "mouthpiece"], "question_type": "what", "dependent_nouns": ["Delta Air Lines", "periodical ,! literature",...], "el_terms": [ "Delta Air Lines": { "wikidata_id": "Q188920", "label": "Delta Air Lines", "aliases": ["DAL", "Delta Air Lines, Inc."},...], ,! ,! ... } There are some cases where terms are typos, so grammatical models should be used to correct them. For each term, we depend on the API searching5 of Wikipedia to x the typos for terms longer than eight characters. We assume that a short term can lead to a wrong search. In some cases, the result may be a term that is more popular than the term we desire. As a result, we may have a wrong x that distorts the meaning of the original sentence. In the example, we apply this API to x the typo moutpiece to mouthpiece.
For the EL methods, we already built APIs for performing ELs using various methods, such as Babelfy, OpenTapioca, Wiki er, and AIDA but we decided to use TagMe, with WAT as an alternative. We gather linked terms to Wikidata which have the link probability higher than 0.9. After that, we apply an oversampling technique over linked terms to increase the number and discrepancy of questions over rare types. We simply replace aliases of linked terms to
their labels to create new questions. In the example, we can produce some new questions like: { "question": "What periodical literature does Delta Air Lines use ,! as a moutpiece?", "new questions": ["What periodical literature does DAL use as a mouthpiece?", "What periodical literature does Delta Air Lines, Inc. use as a mouthpiece?","What literature does Delta Air Lines use as a moutpiece?",...],
We generated more questions by replacing key terms to their roots, such as periodical literature to literature. We assume that by using these replacements, the models will be able to deal with the new data more e ectively.
By using our strategy, we were able to adjust the number of questions and rare types between the original and extended datasets, as shown in Table 1. For each question, we only count single types rather than attening type lists into strings. In comparison to the original dataset, the new datasets have nearly three times more and less in roughly half of rare types than the original dataset. The number of questions and answer types changes slightly when we apply some minor preprocessing measures before the training process. 4.2
According to our initial experiments, we decided to split our datasets into two groups (D1 and D2) each of which will be trained by two BERT models, one for category prediction (also type prediction for the literal and boolean categories) and the other for type prediction of the resource category. In total, we have four BERT models for two datasets (two for DBpedia and two for Wikidata).
{ D1: These datasets are used for category prediction and type prediction for
the boolean and literal categories. They are attened into 5 categories:
boolean, literal-date, literal-number, literal-string, and resource
as seen in the work of Setty and co-authors [
We adapt BERT pre-trained models to a speci c task, in this case text classi cation. We set dropout with p = 0.1 and use linear regression to t hidden size to n classes. In training process, we also use cross-entropy loss and Adam optimization with learning rate = 2e-5. Other hyperparameters are MAX LEN = 192, RANDOM SEED = 42, and BATCH SIZE = 8. We use a small value of BATCH SIZE due to the limited resources on our server. across both DBpedia and Wikidata datasets if the same metric is used. Meanwhile, the numbers of types of Wikidata datasets are now slightly reduced via the oversampling step. 5 5.1
We apply bert-based-cased6 to D1 datasets to predict the categories and types (only for the literal and boolean categories) over questions. As previously stated, the questions are divided into ve categories: boolean, literal-date, literal-number, literal-string, and resource. By this way, the model can predict not only the answer category but also the answer type for questions which belong to the literal category and the boolean category. The only thing left to accomplish is to predict the answer types of questions in the resource category. This task will be completed in the following section.
We split D1 datasets into three subsets: train test, test set, and validation set, in the ratio 8:1:1. After 10 epochs, we will save the best model based on the highest validation accuracy. Since the organizers do not initially provide us with the golden label set, we decide to use validation sets extracted from the original datasets. The previous section discusses type prediction of answers over the literal and boolean categories. In this section, we do the same thing, but for the type prediction of the resource category. We also apply bert-based-cased over D2 datasets for answer types. As with D1, we divided D2 into three subsets: training, testing and validation set with the ratio 8:1:1. Even though we apply an oversampling method to the original dataset, as described in Section 3.2, there are many rare types with fewer than ve questions in the datasets.
Table 5 shows the evaluation metrics on D2 datasets after at least 15 epochs. Our suggestion is to train until train acc is at the highest value as possible to maximize the model's ability to learn the rare types.
Even though we achieve optimistic results from training data steps, we must work on the test sets o ered by the organizers to have the nal evaluations. We consider these sets as holdout sets, providing samples that do not appear mostly in the training data. The category prediction will use accuracy scores as the metric to evaluate our models. The metric lenient NDCG@k (NDCG@5 and NDCG@10) will be applied to evaluate the performance of the type prediction on the DBpedia dataset, whereas the MRR score will be used for the Wikidata dataset.
In Table 6, we obtained the accuracy score of 98.5%, NDCG@5 of 0.727, and NDCG@10 of 0.664 for the DBpedia dataset. When compared to the best performance, our results are among the best and quite competitive. Meanwhile, our method performs best on Wikidata, with an accuracy of 98% and MRR of 0.7 as shown in Table 7. 6
When working with the old datasets, we detect a tiny number of errors relating to null answer types or the improper agreement between questions and answer types. Some questions are too short and can not be analyzed precisely to get semantic units, while others are too long. We consider them all as noises and lter them out before training the data.
Our analysis of question structure is not always correct. In some cases, we are unable to obtain the necessary information, such as the question type, subject, main verb, etc. The parsing method occasionally detects wrong dependent nouns as shown in Table 8. In the rst example, the term 's husband leads to generate a new wrong question "Who is the child of Ranavalona Ihusband?" when replacing 's husband by husband (considered as the noun root). As a result, we must avoid all replacements on possessive nouns containing 's. The second example has no e ect on the replacements, but the parsing cannot split the long term right ascension of malin 1 into two smaller terms, right ascension and malin 1 to produce new questions.
Example 1 question Who is the child of Ranavalona I's husband? key terms "the child", "Ranavalona I", " 's husband " question template Who is fthe childg of fRanavalona Igf 's husband g? Example 2 question Is the right ascension of malin 1 less than 15.1398? key terms " right ascension of malin 1 ", "15.1398" question template Is the f right ascension of malin 1 g less than f15.1398g ? Replacing dependent noun phrases in the creation of new questions does improve the accuracy of category prediction. However, our intuition tells us that we can distort the meaning of questions even in the least aspect. As a result, this may a ect type prediction. We thus avoid applying this task to any dependent noun phrases that are fewer than 8 characters long. We assume that the longer phrases can keep the original meaning better. Since we only take linked terms to Wikidata with those that have a link probability score over 0.9, we may leave a lot of other helpful but equally correct linked terms with lower probabilities.
Due to the occurrence of so many rare types, there are likely not enough rare types in the training set after splitting the data into training, testing, and validation sets. On one side, it helps the model's detection of similar types when the rare types do not appear. On the other hand, this splitting may have an impact on the performance of predicting rare types. Hence, we should place all data on the training set or retrain the model until the accuracy score achieves the best convergence across all sets.
In the DBpedia dataset, the performances are relatively low in comparison to other teams. This happens when we atten answer type lists to strings, instead of having to use di erent models to detect single answer types based on their ranking. 7
In this paper, we participated in Task 1 of the SMART 2021 Semantic Web Challenge, category and type prediction of answers using a set of hint ontologies. We apply spaCy and TagMe to extract sentence components and linked terms from questions. By using a simple oversampling method based on replacements of linked terms and dependent nouns, we are able to expand the size of datasets about three times, targeting to have as many questions as possible, especially on rare answer types. Our project code can be accessed at GitHub 7.
In the experiments, a pretrained BERT model, bert-base-cased, was used to train D1 and D2 datasets to predict answer categories and types. For DBpedia, NDCG@5 and NDCG@10 are 0.727 and 0.664 respectively, with an accuracy of 98.5%. The best results in the Wikidata dataset have an MRR score of 0.7 and an accuracy of 98%. We discover that BERT models perform well in multiclass and multilabel classi cation problems.
In the future, we will improve the analysis parsing of question structure and EL methods to add ontology information on top of the training data. We plan to test various neural networks or hybrid approaches to search for a superior method, as well as try to augment the dataset using entity linking methods and multilingual translation. Finally, the semantic relationships between answer types should be studied by linking to questions in order to minimize the number of types and infer the answer types e ectively.
The work was done with partial support from the Mexican Government through the grant A1-S-47854 of the CONACYT, Mexico, and by the Secretar a de Investigacion y Posgrado of the Instituto Politecnico Nacional, Mexico, under Grants 20211884, 20220859, and 20220553, EDI; and COFAA-IPN. The authors thank the CONACYT for the computing resources brought to them through the
Plataforma de Aprendizaje Profundo para Tecnolog as del Lenguaje of the Laboratorio de Supercomputo of the INAOE, Mexico and acknowledge the support of Microsoft through the Microsoft Latin America PhD Award.