The Knowledge Graph Question Answering (KGQA) systems are required to understand natural language in order to transform textual questions into structured queries to a knowledge graph. One of the important tasks of natural-language understanding (NLU) in the context of KGQA systems is the classi cation of the expected answer type (EAT). In this paper, we present our approach on the EAT classi cation within the SeMantic Answer Type Prediction Task 2021. The approach is based on machine-translation-based data augmentation, it supports 104 input languages1 and works over DBpedia and Wikidata. The obtained evaluation results demonstrate reasonable quality in comparison to both last year's and this year's solutions.
# without EAT prediction SELECT (COUNT(DISTINCT ?obj) as ?count) WHERE {
dbr:Angela_Merkel ?p ?obj . } # ?count = 861 # with EAT prediction SELECT (COUNT(DISTINCT ?obj) as ?count) WHERE { dbr:Angela_Merkel ?p ?obj . ?obj rdf:type ?type .
FILTER(?type = dbo:City) } # ?count = 6 case, the EAT classi cation task can be treated as a multi-class text classi cation task. In the context of the SMART task, the data structure is more sophisticated. There are two class levels: answer category (resource, literal, boolean) and answer type. Hence, the class taxonomy is not at and requires approaches for hierarchical classi cation.
The o cial description of the data states4: If the category is \resource", answer types are ontology classes from either the DBpedia ontology5 or the Wikidata ontology6. If the category is \literal", answer types are either \number", \date", or \string". For the category \boolean", no additional specialization is de ned. The number of unique \resource" classes is high. Moreover, they are represented in the form of a list.
While following our long-term research agenda of increasing the
accessibility of KGQA systems and their components through multilingualization, the
presented solution is mainly based on the multilingual language models and
datasets (as in our last year iteration [
This work is structured as follows, in Section 2 we review the related research, Section 3 describes the exploratory data analysis of the provided datasets, we describe our approach in Section 4 and present the evaluation results in Section 5. Section 6 concludes the paper. 2
The entity- and type-centric models were introduced in [
There are two class levels in the datasets: answer type category (resource, literal, boolean) and answer type. As the class taxonomy is not at it requires us to use approaches for hierarchical classi cation. The o cial description of the data states7: If the category is \resource", answer types are ontology classes from either the DBpedia ontology8 or the Wikidata ontology9. If the category is \literal", answer types are either \number", \date", or \string". For the category \boolean" no additional specialization is de ned. Such data has to be analyzed on di erent levels for class distribution, noise, missing values, etc. In this section, we demonstrate the analysis of both DBpedia and Wikidata answer type classi cation datasets. 3.1
DBpedia Dataset The DBpedia dataset contains Train (40,621 examples) and Test (10,093 examples) subsets. After removing the null values from the Train subset, the number of examples decreased to 37,061 (-3,560 examples). The example of the DBpedia dataset for the answer type prediction is shown in Figure 2. Here, if the category is resource, the answer type eld is represented as an ordered list of DBpedia ontology types, where the rst type is the most speci c and the last is the most general one (see the very rst question in the Figure). While manually analyzing this year's SMART dataset we were capable to nd several noisy examples (see Figure 3). In this example, the type eld contains duplicated DBpedia ontology types.
We also analyzed the distributions of values for category, literal, type elds. The results are demonstrated in Figure 4. There is a huge imbalance towards the resource value while the literal values are more or less balanced. The distribution of the resource eld values is demonstrated in Figure 5. Surprisingly, the most frequent values of the answer type eld contain noise. For example, the second most frequent value has incorrect order of DBpedia types. The third most frequent value has DBpedia types of di erent hierarchies (dbo:Place and dbo:Agent). 7 https://smart-task.github.io/ 8 http://mappings.dbpedia.org/server/ontology/classes/ 9 https://www.wikidata.org/wiki/Wikidata:WikiProject_Ontology "id": "dbpedia_1", "question": "Who are the gymnasts coached by Amanda Reddin?", "category": "resource", "type": ["dbo:Gymnast", "dbo:Athlete", "dbo:Person", "dbo:Agent"] "id": "dbpedia_2", "question": "When did Margaret Mead marry Gregory Bateson?", "category": "literal", "type": ["date"] "id": "dbpedia_3", "question": "Is Azerbaijan a member of European Go Federation?", "category": "boolean", "type": ["boolean"] "id": "1", "question": "Who is the child of Ranavalona I s husband?", "category": "resource", "type": ["person", "omnivore", "natural person"] The Wikidata dataset contains Train (43,604 examples) and Test (10,864 examples) subsets. We also cleaned the training data by removing the null values. Consequently, the number of examples decreased to 43,554 (-50 examples). In this aspect, the Wikidata data has signi cantly fewer null values rather than the DBpedia data. An example of the Wikidata dataset for the answer type prediction is shown in Figure 6.
Here, if the category is resource, the answer type eld is represented as a list of Wikidata classes that are retrieved according to the following SPARQL query: PREFIX wd: <http://www.wikidata.org/entity/> PREFIX wdt: <http://www.wikidata.org/prop/direct/> SELECT ?subClasses WHERE { wd:Q123456789 wdt:P31 ?x . # subject to be replaced with actual answer entity ?x wdt:P279 ?subClasses . }
While manually analyzing the training subset of Wikidata we were capable to nd several noisy examples of di erent nature in comparison to DBpedia (see Figure 7). In this example, the type eld contains duplicated DBpedia ontology types.
We also analyzed the distributions of values for category, literal, type elds. The results are demonstrated in Figure 8. For Wikidata we observed the same data distribution patterns: there is a huge imbalance towards the resource "id":10395, "question":"what is the grammatcal mood of turkish", "category":"resource", "type":[
"grammatical category", "Q26869183", "grammatical mood"
value while the literal values are more or less balanced. The distribution of the
resource eld values is demonstrated in Figure 9. Obviously, the answer type
values are extremely imbalanced towards the ones related to the person class.
We assume that the observed noise in the data was unintended and there was a
risk of having the data quality in the test dataset. Hence, we decided not to use
this year's SMART Task data. In our local training and evaluation process, the
data from the previous year was used [
While following our long-term research agenda on enhancing multilingual
accessibility of KGQA systems [
We used a multi-level hierarchical classi cation pipeline for both DBpedia and Wikidata. The pipeline consists of the following models: (1) category classi er, (2) literal classi er { are common models for both knowledge graphs, and (3) resource classi er { trained separately for DBpedia and Wikidata. The architecture of the classi cation pipeline is shown in Figure 10. The resource classi er for DBpedia was trained in the multi-class classi cation setting to predict the most speci c type of the hierarchy. When the prediction is executed, the rest of the hierarchy is fetched from DBpedia via the following SPARQL query based on the predicted type: PREFIX owl: <http://www.w3.org/2002/07/owl#> PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> PREFIX dbo: <http://dbpedia.org/ontology/> SELECT DISTINCT ?parentClass WHERE { <predictedType> rdfs:subClassOf* ?parentClass .
FILTER(?parentClass != <predictedType> && CONTAINS(STR(?parentClass), "dbpedia")) }} The resource classi er for Wikidata was trained in the multi-label classi cation setting. It works without any additional steps in the prediction phase.
0.991 0.643 0.577
0.980 0.430
For the classi cation model, BERT multilingual model [
The prediction results were evaluated with the following metrics. For both
DBpedia and Wikidata, the quality of the category predictions was measured
using accuracy. The quality of the answer type predictions was measured using
Mean Reciprocal Rank (MRR) for Wikidata and lenient Normalized Discounted
Cumulative Gain @k with a Linear decay (NDCG@k) for DBpedia, where k 2
5; 10 [
The evaluation was done by organizers on the private dataset using an internal process. The nal results are shown in Table 1. The obtained results are demonstrating reasonable quality in comparison to the other participants as well as to the results of the last year. The Accuracy score of the solution demonstrates one of the best results on both DBpedia and Wikidata, while metrics related to the ranking of the answer type hierarchies (NDCG@k, MRR) achieved relatively poor results. The source code of our solution is available online11. In addition, we also have deployed a demo-interface for EAT classi cation over DBpedia online12. 6
We would like to raise the following questions concerning the evaluation process. First, as we observed a signi cant imbalance in the data w.r.t. the category values (see Section 3), we think that the usage of accuracy score to measure 10 https://huggingface.co/bert-base-multilingual-cased 11 https://github.com/Perevalov/smart-2021 12 https://webengineering.ins.hs-anhalt.de:41009/eat-classification the results is not the best option since that it is not robust to the imbalanced data. Instead, we propose to use precision and recall scores computed in the classi cation setting. Secondly, as the answer types of the resource category questions in Wikidata are not ordered and do not form a hierarchy, we think that the usage of mean reciprocal rank is not acceptable as this metric is used to evaluate ordered result sets. Hence, the measure for unordered lists is naturally applicable to this task, such as precision and recall calculated in an informationretrieval setting.
In this paper, we demonstrated our approach for hierarchical EAT prediction
based on a multi-level classi cation pipeline. As we used multilingual data and
models for training, the classi cation pipeline supports input in 104 languages.
The evaluation process demonstrated reasonable results w.r.t. the quality
metrics. For future work, we are targeting on improving the quality of the resource
answer types classi cation, and we are planning to create a study on the impact
of EAT classi cation on QA quality for multiple QA systems using a
componentoriented QA framework (e.g., [