This paper describes the CitySAT system that we developed for the DBpedia Answer Type (AT) prediction task of the SMART 2021 challenge. The challenge can be interpreted as a multi-class classification task that takes natural language questions and returns pairs of the predicted answer category and types. For training, we merged the SMART 2021 DBpedia dataset with the 2020 dataset given for the previous year's AT task. In this study, three local Machine Learning (ML) models are deployed to cover the three diferent types of task and question (category prediction, literal type prediction and resource type prediction). The best model obtains a 98.36% accuracy for the category prediction using a Logistic Regression (LR) classifier. Similarly, another LR model results in 97.90% accuracy for the literal type prediction task. Lastly we also built a Multi-Layer Perceptron (MLP) model to deal with several ontology classes (∼ 760 classes for DBpedia) in the resource type prediction task. The best MLP model achieves 79.34% on the merged training dataset. The final system output obtained a 98.4% accuracy, 84.2% NDCG@5, and 85.4% NDCG@10 on the (oficial) test dataset.
In computer science, Answer Type prediction (AT) is a research domain providing with simplified tasks of the Question Answering (QA) discipline. It inherits the QA task missions to understand the meaning of natural language questions but identifies their answer types instead of retrieving the most relevant information among the answer candidates. Due to the number of classes of the categorical answer type candidates, AT task can be interpreted as a multi-class text classification task upon pre-defined classes.
The SeMantic Answer type and Relation prediction Task (SMART),1
organized by the International Semantic Web Conference (ISWC), expands the AT
⋆ Copyright ©2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
1 https://smart-task.github.io/
challenge to a more complicated structure of knowledge base data which
consists of the answer category and type in two hierarchical levels. They provide
two versions of large-scale dataset for the AT task; one edition using DBpedia
ontology (∼ 760 classes) and the other edition using the Wikidata taxonomy
(∼ 50K classes). This paper demonstrates our participation in the SMART 2021
AT task [
The CitySAT system firstly verifies the accuracy score of the answer category prediction and additionally uses the lenient Normalized Discounted Cumulative Gain (NDCG) at each 5 and 10 answer type values to compete with comparable systems. The best optimization confirms 98.4% accuracy, 84.2% NDCG@5, and 85.4% NDCG@10. As NDCG evaluation metric follows a linear decay, NDCG@10 is highly valued and our result proves to the beneficiaries that the more predeifned knowledge the better the system predicts.
The rest of the paper is structured as follows. Section 2 (Context) surveys the used materials and related work. Section 3 (Methods) presents a detailed view of the entire research progress from data loading to evaluation with justifications for each of the steps. Section 4 (Results) summarizes the respective high-valued experiments. Section 5 (Discussion and Future work) examines what is a meaningful correlation between our research motivation and results that conclude the next stage of studies. The code for reproducing the experimental results of this study is publicly available at https://github.com/chaeyoonyunakim/ smart-2021-AT. 2
SMART 2020 [
To start with training datasets, the previous work (e.g., [
Another one of the most important observations over the previous work is that they presented a very high accuracy for their answer category classifier on the upper level of the hierarchical data structure. For the 2020 performance results, Table 2 shows that top ranked outputs even exceeded over 90% accuracy. It led an initial decision that our experiments can be conducted on traditional Central Processing Units (CPUs). Due to the increased data volume, however, it is worth to define how to selectively adapt the referencing models in this study. Hence, some initial data analysis on lower level hierarchy (i.e., targets on the answer type) is following in next section. 2.1
Regarding the shape of the bottom level data for answer type, Figure 1 gives
details how many ontological classes are located in the resource type. Both 2020
and 2021 DBpedia editions are mostly consisted of equal or less than 6 answer
types, but the distribution of type value numbers is more positively skewed in the
2020 edition. Therefore, the previous systems considered the first five (e.g., [
To understand the context of the answer type, Figure 2 illustrates an example
question and how its ontological classes are lined up in the given dataset. Among
the classes, there is an exception (dbo:Location) which is defined as an entity
in DBpedia but it is considered as a class having an equal level with dbo:Place
specifically in SMART 2020 and SMART 2021 shared tasks. As explained in [
In SMART AT task, the training dataset and test dataset share the same JSON format (see example below). Comparing the DBpedia datasets from the two SMART editions, at first glance, we found the “id” attribute changed its value from “dbpedia 1” to “1” and this has been taken into account when merging the datasets according to the arrangement in the 2021 edition. { "id": "1", "question": "Who are the gymnasts coached by Amanda Reddin?", "category": "resource", "type": ["dbo:Gymnast", "dbo:Athlete", "dbo:Person", "dbo:Agent"] }
This study uses Python pandas library [
To construct a robust data structure, feature engineering is necessary to drop the missing value observations such as zero number of type values in resource category and invalid values such as “n/a” in question. Due to the NDCG evaluation at the 10th answer type, we justify outliers as having more than 10 components in the answer type. In the case of not having the counter number of components, an indicator of missing value has to be marked since the evaluation matrix will return infinite distance of type path that means no relevance between the output (i.e., prediction types) and the ground truth (i.e., gold types).
As the ground truth was not open for the 2021 challenge, this study used
20% of the merged training dataset for validation to check the performance of
the system. For submission, the test dataset (9,104 questions) is predicted by
the best model trained with the entire training dataset (39,556 questions and
answers).
This study uses the NLTK library [
Initially, this study was specifically interested in dealing with Wh-terms (Who, What, When, Where, Which, Whom, Whose, Why, and include How) questions which accounts for 84.4% of the training questions. So, we customized stop words dictionaries to exclude the Wh-terms. Unfortunately, however, the new stop-words removal works less eficiently in despite of the initial interests. Hence, this study keeps the original NLTK stop words for further analysis.
Additionally, this study explores term-frequency (TF) and inverse
termfrequency (TF-IDF) in text feature extraction using CountVectorizer and
TFIDFVectorizer from the scikit-learn library [
Categorical targets (answer category and type) are mapped into numerical labels. In particular, we distribute the answer type by wish-number of training target classes and encode “missing” if nothing exists in the location. For example, Figure 5 illustrates a sample conceptual data when we select the 10 values of the answer type across all categories. The classifiers can be programmed to have the first and single value (i.e., type1 column in Figure 5) if the category is either “boolean” or “literal”. However, one or more values are taken when the category is “resource”. The order of values is mixed as stated in Figure 2, and our justification is to select the first five to ten values.
Furthermore, the selective number of features are mapped into a dictionary
with keys for an indicator of the value’s location. For example, by setting the
argument type no 11 (i.e., type1 to type10) in the reference code below,
dictionary keys [“type1”, · · · , “type10”] and dictionary values [{“dbo:Opera”: 0,
· · · “dbo:RadioStation”: 297}, · · · , {“dbo:Politician”: 0, · · · , “dbo:Entomologist”:
21}] will create 10 json files for {“type1”: {“dbo:Opera”: 0, · · · “dbo:RadioStation”:
297}} to {“type10: {“dbo:Settlement”: 0, · · · , “dbo:Village”: 21}}. The final
accumulated dictionary of type maps is {“type1”: {”dbo:EducationalInstitution”:
0, · · · , “dbo:Holiday”: 282}, “type2”: {“dbo:MusicalWork”: 0, · · · “dbo::Presenter”:
171}, · · · , “type10”: {“dbo:Politician”: 0, · · · , “dbo:Entomologist”: 21}}.
To perform hierarchical classification, this study considers local classifiers per
level as shown in Figure 6 which is widely used in the state-of-the-art (e.g., [
To decide a suitable classifier for each level, we implemented a small
sample batch of python codes with several Machine Learning (ML) algorithms by
importing SVM, LogisticRegression (LR), and MLPClassifier (MLP) from
scikitlearn [
CitySAT system is programmed in Google Colab CPU environment with two processing threads. The evaluation of each stage algorithms has been conducted with validation data which is 20% of the training dataset.
By expanding experiments from Section 3.3, we get the best optimised hyperparameters in a combination system of two LRs and a MLP as briefly captured in the Appendix. Figure 7 shows the results of the design of our classification model from top to bottom level, starting from a single LR model which classifies the answer category at the top level. Two diferent models are used to classify the answer type at the bottom level: a LR model for the type of the literal category and a MLP model for the type of the resource category.
As last step, to meet the submission format specifications, it is essential to decode all mapping data and convert the format back to the JSON format as shown by the JSON below. The CitySAT system workflow is depicted in Figure 8. { "id": "5586", "category": "resource", "type": ["dbo:Company", "dbo:Activity", "dbo:RecordLabel", "dbo:Agent", "dbo:Species", "dbo:Organisation", "dbo:AdministrativeRegion", "dbo:Location", "dbo:Country", "dbo:PopulatedPlace"] } other two scores (Accuracy, NDCG@5) are similar. The best results of CitySAT is ranked at the top in DBpedia AT task.
With the given DBpedia data in the SMART 2021 AT challenge, this study tried various explorations on text normalization. Including filters of Wh-terms in stop words, there were multiple congfiuration settings we could have imagined to have improvement in classification performance which was not the case for this challenge. This opens the door for future work in finding how to improve the text features involving semantic meanings in a more human thought process.
Although CitySAT models are optimised in Section 3.4 to find the best combination of local classifiers for two levels, there might be more options and diferent combinations of models that can be discovered in the future. Especially, once we expand our evaluation environment to a Graphics Processing Unit (GPU), there are more applicable ML models for our problem.
As in previous studies, we have also found that several of the participating
systems used the fine-tuned BERT models [
We would like to thank the ISWC conference and the SMART challenge organisers. This work was partially supported by the SIRIUS Centre for Scalable Data Access (Research Council of Norway). Appendix # distribute categorical targets (answer category & type) to numericals def type_to_int(self, data, type_no): return data.type.map( lambda x: self.type_maps[f"type{type_no}"][x[type_no - 1]] if len(x) >= type_no else self.type_maps[f"type{type_no}"]["missing"] )