This paper presents Sherlock, a semi-automatic quiz generation system for educational purposes. By exploiting semantic and machine learning technologies, Sherlock not only o ers a generic framework for domain independent quiz generation, but also provides a mechanism for automatically controlling the di culty level of the generated quizzes. We evaluate the e ectiveness of the system based on three real-world datasets.
Interactive games are e ective ways of helping knowledge being transferred
between humans and machines. For instance, e orts have been made to unleash
the potential of using Linked Data to generate educational quizzes. However,
it is observed that the existing approaches [
Moreover, a system that can generate quizzes with di erent di culty
levels will better serve users' needs. However, such an important feature is rarely
o ered by the existing systems, where most of the practices simply select the
distractors (i.e., the wrong candidate answers) at random from an answer pool (e.g.,
obtained by querying the Linked Data repositories). Some work has attempted
to determine the di culty of a quiz but still it is simply based on assessing the
popularity of a RDF resource, without considering the fact that the di culty
level of a quiz is directly a ected by semantic relatedness between the correct
answer and the distractors [
In this paper, we present a novel semi-automatic quiz generation system (Sherlock) empowered by semantic and machine learning technologies. Sherlock is distinguished from existing systems in a few aspects: (1) it o ers a generic framework for generating quizzes of multiple domains with minimum human e ort; (2) a mechanism is introduced for controlling the di culty level of the generated quizzes; and (3) an intuitive interface is provided for engaging users
Similarity Computation
LOD Similarity
Adaptive
Clustering
Template-based Question and Answer Generator
Incorrect Distractor Database Question and Answer Database
Online
Quiz Renderer
Quiz Creator in creating customised quizzes. The live Sherlock system can be accessed from http://sherlock.pilots.bbcconnectedstudio.co.uk/1. 2
1 For the best experiences, please use Safari or Opera to access the demo. educational background). Furthermore, to enhance a user's learning experience, the \learn more" link on the bottom left of the interface points to a Web page containing detailed information about the correct answer (e.g., Cheetah). Quiz Creator: Fig. 2(b) depicts the quiz creator module, which complements the automatic quiz generation by allowing users to create customised quizzes with more diverse topics and to share with others. Quiz authoring involves three simple steps: 1) write a question; 2) set the correct answer (distractors are suggested by the Sherlock system automatically); and 3) preview and submit. For instance, one can take a picture of several ingredients and let people guess what dish one is going to cook. The quiz creator interface can be accessed from http://sherlock.pilots.bbcconnectedstudio.co.uk/#/quiz/create. 3
This demo aims to show how Sherlock can e ectively generate quizzes of di erent domains and how well a standard similarity measure can be used to suggest quiz di culty level that matches human's perception. The hypothesis is that if some objects/entities have higher degree of semantic relatedness, their di erences would be subtle and hence more di cult to be disambiguated, and vice versa.
We investigated the correlation between the di culty level captured by the similarity measure and that perceived by human. To test our hypothesis, a group of 10 human evaluators were presented with 45 testing quizzes generated by Sherlock based on the BBC Wildlife domain data, i.e., 15 quizzes per di culty level. Next the averaged pairwise similarity between the correct answer and distractors of each testing quiz were computed, as shown in Fig. 3(a). Fig. 3(b) demonstrates that the quiz test accuracy of human evaluation indeed shows a negative correlation (r = 0:97, p < 0:1) with the average similarity of the quiz answer choices (i.e., each datapoint is the averaged value over 15 quizzes per di culty level). This suggests that LDSD is an appropriate similarity measure for indicating quiz di culty level, which inlines with our hypothesis.
In another set of experiments, we evaluated Sherlock as a generic framework for quiz generation, in which the system was tested on structural RDF datasets from three di erent domains, namely, BBC Wildlife, BBC Food and BBC YourPaintings2, with 321, 991 and 2,315 quizzes automatically generated by the system for each domain respectively. Bene ting from the domain-independent similarity measure (LDSD), Sherlock can be easily adapted to generate quizzes of new domains with minimum human e orts, i.e., no need to manually de ne rules or rewrite SPARQL queries. 4
In this paper, we presented a novel generic framework (Sherlock) for generating educational quizzes using linked data. Compared to existing systems, Sherlock o ers a few distinctive features, i.e., it not only provides a generic framework for generating quizzes of multiple domains with minimum human e ort, but also introduces a mechanism for controlling the di culty level of the generated quizzes based on a semantic similarity measure.
Acknowledgements The research described here is supported by the BBC Connected Studio programme and the award made by the RCUK Digital Economy theme to the dot.rural Digital Economy Hub; award reference EP/G066051/1. The authors would like to thank Ryan Hussey, Tom Cass, James Ruston, Herm Baskerville and Nava Tintarev for their valuable contribution. 2 http://www.bbc.co.uk/nature/wildlife, http://www.bbc.co.uk/food and http: //www.bbc.co.uk/arts/yourpaintings