In this paper, we engage the Task 2 of the SMART Task 2021 challenge in predicting relations used to identify the correct answer of a given question. This is a subtask of Knowledge Base Question Answering (KBQA) and o ers valuable insights for the development of KBQA systems. We introduce our method, combining BERT and data oversampling with text replacements of linked terms to Wikidata and dependent noun phrases, in predicting answer relations in two datasets. For the DBpedia dataset, we obtain F1 of 83.15%, precision of 83.68%, and recall of 82.95%. Meanwhile, for the Wikidata dataset we achieved F1 of 60.70%, precision of 61.63%, and recall of 61.10%.
In Natural Language Processing (NLP), Knowledge Base Question Answering (KBQA) is a task that deals with answering questions using the relevant information provided in the knowledge base (KB). Natural language questions are converted into SPARQL queries to retrieve answers from KBs. The question types can vary, depending on the targeted problem to nd the answers, such as simple questions need to have small snippets of text, complex questions require inferencing and synthesizing information, or long questions which are more difcult to interpret, etc. NLP researchers normally build pre-de ned templates to generate questions or use crowdsourcing to produce the desired questions.
To correctly map questions to relevant KB relations, relation linking is an
important task for improving signi cantly the performance of question
answering. It has been a challenging problem for NLP researchers due to multiple
and implicit relations in questions, limited annotated training data, and
lexicalsemantic di erences [
In this paper, we participate in Task 2 { relation set prediction of the SMART
Task 2021 3 over DBpedia and Wikidata datasets [
We applied a combination of BERT models and data oversampling by text replacements of linked terms to Wikidata and dependent noun phrases to solve the problem. Besides this section, the other sections follow this structure: Section 2 explain the background of KBQA and techniques used to improve the problems in the eld. Section 3 and Section 4 describe the datasets and our methodology used to train the models and produce the results. Finally, we present our experiments and error reports, as well as conclusions and future works, in Sections 5 and Section 6.
The task of relation set prediction requires a thorough understanding of the
KBQA background. Question answering has evolved from simple QA by
achieving signi cant results to complex QA tasks. Some popular datasets related to the
task are Question Answering over Linked Data (QALD) [
There have been several methods proposed for complex question answering
that can be listed as Information Retrieval (IR) based methods, Neural
Semantic Parsing based methods, and traditional methods. Traditional methods
mainly rely on template-based models [
Some previous approaches to identify relation linking has been using semantic
parsing [
KBQA corpora are usually imbalanced due to they consists of numerous
natural language questions, created from language diversity and human creativity. To
help the dataset more balanced or less biased, oversampling and undersampling
techniques are usually applied to reduce popular data and increase rare data. For
oversampling, there are many techniques such as SMOTE [
Referring to some studies [
Next, for each question, we split its relation list into single relations, then form the distribution of relations over questions as in Figure 2. If the number of relations that appear are in less than 5 questions, we call them rare relations. We have 226 and 2299 ones corresponding to DBpedia and Wikidata datasets. It is clear that the datasets are imbalanced and contain many rare relations, thus this becomes a challenge for not only this paper but also for text classi cation.
We rst thought about reducing the number of relations by depending on the ontology hierarchy structure or removing all rare types. However, the former takes time to analyze and the latter might a ect the outcome performance in general. Therefore, we decided to apply an oversampling technique by text replacements of linked terms to Wikidata and dependent noun phrases to reduce the number of rare types as many as possible.
For each question, we used spaCy4 v.2.3.2 to analyze the question structure and to get its components, such as question type, subject, main verb (also ROOT), terms (noun phrases, dependent noun phrases) in order to build the sentence template, and apply entity linking (EL) methods to extract terms connecting to Wikidata. We take some spaCy components such as en core web lg, STOP WORDS, lemmatizer (Lemmatizer, ADJ, NOUN, VERB), and sentencizer pipeline. The sentence template is built by a greedy algorithm which absorbs all longest terms. The text below (in json format) represents the structure analysis of a random example. { "question": "What is the safety classification and labeling for ,! hydrochloric acid?", "relations": ["P4952"], "relation_labels": ["safety classification and labelling"], "question_template": "What is {the safety classification} and ,! {labeling} for {hydrochloric acid}?", "key_terms": ["labeling", "hydrochloric acid"], "subject": ["the safety classification"], "main verb": ["is", -1], "aux_verb": ["is", 1], "entities": ["labeling", "hydrochloric acid", "acid"],
"question_type": "what", "dependency_nouns": ["hydrochloric acid", "hydrochloride", ,! "acid",...], "el_terms": [ "hydrochloric acid": { "wikidata_id": "Q2409", "label": "Hydrochloric acid", "aliases": ["HCl", "muriatic acid"},...],
We found some terms containing typos and started to think how to correct them by using a grammar model. However, this approach may create more extra works. Instead, we deploy a simple method, API searching 5 of Wikipedia. For each mapped term no matter it has typos or not and longer than 8 characters, we used the API searching to x typos may have. If the new term is the same with the original term, there is no need x anything here. Otherwise, if the length of new term is equal or larger 1 or 2 than the old one, we will get this term. We assume the longer phrases can keep the original meaning better. However, this method is not always stable when the result could be a term that is more popular than the term we want.
For the EL, we already built APIs 6 for doing EL in several other methods,
such as Babelfy, OpenTapioca, Wiki er, and AIDA but decided to use TagMe
API due to its availability on D4Science.org 7. Also on the same website, WAT
API 8 is a better method based on TagMe but works only with English [
In fact, we even produced more questions by replacing key terms by their roots, such as safety classification to classification. We assume these
text replacements can help the models to deal better with the new data, since modi ers are used to carry less lexical information than headers of phrases. Table 2 shows the changes between the original datasets and extended datasets by our oversampling method. Compare to the original datasets, the new DBpedia dataset has about 2 times more questions and less than a roughly half rare relation. Meanwhile, about 5 times more questions and less than roughly a quarter of rare relations are results in the Wikidata dataset. 4.2
We applied bert-based-cased as a pretrained BERT model for the training process to see how well BERT can deal with relation set prediction from input questions. Considering relation set prediction is a problem of text classi cation, we thus atten relation lists into strings in both DBpedia and Wikidata datasets to easier train. For example, the relation list [['dbo:director'],['dbp:awards ']] of the question "What are the awards won by the film director of Saraband ?" will be attened as string dbo:director;dbp:awards. The delimiter \;" refers to a divider between two relations.
Table 3 indicates the statistics of atten relations over DBpedia and Wikidata datasets. We only take the rst item of each relation list for the attening step to reduce the number of produced relation strings. For clear, this means each question has only one relation. As far as we know, according to the evaluation code 9, this may a ect the nal performance in general. However, there is
no guarantee that the performance is better if we keep all relation lists for the training. We will clarify this in the future works. Compare to Table 2, the numbers of relations and rare relations now change signi cantly, which hint us about the di culty that models have to deal with. 5
For each attened dataset, we split it into 3 subsets, training, validation, and test sets with the ratio 8:1:1. Within 15 epochs, the best model will be saved with the highest validation accuracy. In the Wikidata dataset, the training values are lower than those in the DBpedia dataset because it contains more relations. In Table 4, the validation accuracy of 0.84 is acceptable, but it suggests us the need to train the model more. After the training process, we validated our models with test and golden label sets provided by organizers to have nal evaluations. For each dataset, the average values of precision, recall, and F1 metrics are applied to all questions.
Table 5 shows our performance on the ranked table o ered by organizers. Unfortunately, there are only 2 teams participating in the challenge. Compare to the other team, we have similar results on the DBpedia dataset, while on the Wikipedia dataset, we have a lower performance. This may be from a gap between the validation accuracy (0.84) and the training accuracy (0.89), which are also not expected scores in our training process.
In future, we should use all data in the training process instead of splitting into di erent sets, or train the model until meeting the smallest gap between accuracies in all sets. The performance of both teams recon rms the task di culty as declared by the organizers.
We see some minor errors in the dataset, but they do not a ect the outcome performance in general. However, our text replacement method contains an error. The question "What is the Beethoven's piano sonatas?" will produce a new question as "What is the Beethovensonatas?" when replacing 's piano sonatas to its root sonatas. Therefore, we have to avoid all replacements on the possessive nouns containing 's. Besides, we will improve our parsing analysis because we can not extract the correct components from sentences in some cases. 6
In this paper, we participate in Task 2 of the SMART 2021 Semantic Web Challenge, relation set prediction. We applied spaCy and TagMe to extract sentence components and linked terms from questions. By using a simple oversampling method based on text replacements of linked terms to Wikidata and dependent nouns, we were able to expand the size of datasets, targeting to have a higher number of questions as many as possible, especially on rare answer relations.
In the experiments, a pre-trained BERT model, bert-base-cased is used for the training process on atten datasets to predict relations. For the DBpedia dataset, precision and recall are 83.68% and 82.95% while F1 is 83.15%. We obtained lower metric values for the Wikidata dataset with the precision of 61.63%, recall of 61.10%, and F1 of 60.70%.
In the future, we will improve the analysis parsing of question structure and EL methods to add ontology information on top of the training data. We will also try with other neural networks or any hybrid approach to search for a better method, as well as try to augment the dataset by other entity linking methods and multilingual translation. At last, the semantic relationships between relations should be studied in linking to questions to minimize the number of relations and infer relations e ectively.
The work was done with partial support from the Mexican Government through the grant A1-S-47854 of the CONACYT, Mexico, grants 20211784, 20211884, and 20211178 of the Secretar a de Investigacion y Posgrado of the Instituto Politecnico Nacional, Mexico. The authors thank the CONACYT for the computing resources brought to them through the Plataforma de Aprendizaje Profundo para Tecnolog as del Lenguaje of the Laboratorio de Supercomputo of the INAOE, Mexico and acknowledge the support of Microsoft through the Microsoft Latin America PhD Award.