Microbenchmarks are used to test the individual components of the given systems. Thus, such benchmarks can provide a more detailed analysis pertaining to the di erent components of the systems. We present a demo of the QaldGen [5], a framework for generating question samples for micro benchmarking of Question Answering (QA) systems over Knowledge Graphs (KGs). QaldGen is able to select customised question samples from existing QA datasets. The sampling of questions is carried out by using di erent clustering techniques. It is exible enough to select benchmarks of varying sizes and complexities according to userde ned criteria on the most important features to be considered for QA benchmarking. We evaluate the usability of the interface by using the standard system usability scale questionnaire. Our overall usability score of 77.25 (ranked B+) suggests that the online interface is recommendable, easy to use, and well-integrated7.
General-purpose benchmarks are designed to test the overall performance of the system. These benchmarks evaluate the overall performance of the system by providing tests and performance metric. On the other hand, microbenchmarks are more speci c, designed to test ne-grained components of the complete system [1]. The tests and performance metric used in such suggested benchmarks is more speci c to the components on that is being tested. The proposed benchmark provides more detailed, use-case speci c, and component-level evaluations to pinpoint the pro and cons of the system which cannot be achieved from the results of the general-purpose benchmarks.
Various QA datasets such as LC-QuaD8 and QALD9 have been used to evaluate the performance of QA systems over RDF knowledge graphs. In these general-purpose benchmark evaluations, the global metrics of precision, recall, and F-score are used as a performance indicator. Although informative, these evaluations do not shed light on the strength and weakness of a particular component of the QA system. Furthermore, various benchmarks features such type of the benchmark questions (e.g., what, who etc.) and the corresponding answers (e.g. boolean, count, list etc.), the number of entities in the question, the number of triple patterns and joins in the corresponding SPARQL queries etc. have a signi cant impact on the performance of the existing QA systems [3,4]. For instance, the overall winner of the 6th edition of the Question Answering over Linked Data Challenge (QALD6) was CANALI, which su ered limitations when the question started with "Give me". CANALI is outperformed by another QA systems UTQA for such type of questions [3], the problems were highlighted as a result of performing micro analysis on QA system.
To ll this gap, we propose QaldGen [5], a framework for automatic selection of components-level microbenchmarks for QA systems over knowledge graphs. The framework is able to generate question samples customized by for a user in terms of the di erent QA-related important benchmarks features. The framework generates the desired question samples from existing QA datasets (LC-QUAD and QALD9) by using di erent clustering methods, while considering the customized selection criteria speci ed by the user. 2 2.1
As mentioned before, our framework creates question samples for micro benchmarking from existing well-known QA over KGs datasets LC-QUAD and QALD9. LC-QuAD contains a total of 5000 questions while QALD9 contains a total of 408 questions which also include questions from QALD1-QALD8. We automatically annotate total 5408 questions from QALD9 and LC-QuAD datasets with 51 important QA related features. We then convert the annotated questions into RDF format and name the resulting RDF datasets as QaldGenData. This dataset can be reused in training machine learning approaches related to question answering. 2.2
The benchmark generation is carried out the following four main steps: (
8http://lc-quad.sda.tech/ 9http://qald.aksw.org/ The online demo and source code of the QaldGen is available at the QaldGen homepage http://qaldgen.aksw.org/. Figure 1 shows the online interface of the QaldGen, which is comprised of ve main steps: 1. Selection of clustering method: The rst step is to select the question sample generation method(s). Currently, our framework supports 6 wellknown clustering methods namely DBSCAN+Kmeans++, Kmean++, Agglomerative, Random selection, FEASIBLE and FEASIBLE-Exemplars. 2. Parameters selection: The second step is the selection of clustering methodrelated parameters like the number of queries in the resulting benchmark etc. 3. Question Sample personalization: The third step allows to further customize the resulting question sample which can be used for micro benchmarking QA systems. This can be done by using a single SPARQL query. 4. Results: The diversity score and the similarity errors for the selected methods will be shown as bar graphs. 5. Question Sample download: The resulting micro benchmarks can be nally downloaded to be used in the evaluation. 4
An evaluation of the QaldGen can be found in [5]. To assess the usability of our system, we have used the standardized, ten-item Likert scale-based System I needed to learn a lot of things before I could get going with this system (10)
I felt very confident using the system (9)
I found the system very cumbersome to use (8) I would imagine that most people would learn to use this system very quickly (7)
I thought there was too much inconsistency in this system (6)
I found the various functions in this system were well integrated (
I thought the system was easy to use (
I found the system unnecessarily complex (
Usability Scale (SUS) [2] questionnaire10 which can be used for global assessment of systems usability. The survey was posted through Twitter with the ISWC conf hashtag and was lled by 20 users.11 The results of SUS usability survey is shown in Figure 2. We achieved a mean usability score of 77.2512 indicating that the online interface is recommendable, easy to use, and well-integrated.
This work has been supported by the project LIMBO (Grant no. 19F2029I), OPAL (no. 19F2028A), KnowGraphs (no. 860801), Science Foundation Ireland (Grant no. SFI/12/RC/2289), and SOLIDE (no. 13N14456) 10Our survey can found at: https://forms.gle/etaUcgRHHH3ima9u5 11As of June 28th, 2019. Responses summary: https://bit.ly/2XfMy1y 12Please see https://bit.ly/2xiukgl for the interpretation of the score