We propose a method to generate quizzes on a museum electronic guide application. While a museum is considered to be a place for learning, it would be hard for a visitor to actively appreciate the exhibits in the museum, especially when they have little knowledge about the exhibits. In this study, we develop a method that automatically generates quizzes about the exhibits on an electronic guide application. The proposed method utilizes a BERT model that is trained with the additional corpus constructed from the descriptions of the exhibits and automatically generates a quiz about exhibit. By solving quizzes about the exhibits during the museum visit, we expect that a visitor's museum experience would be more active, and they would understand the exhibits more deeply. We implement the proposed method on the electronic guide application that is designed for the National Museum of Ethnology, Japan (a.k.a. Minpaku).
There are many museums around the world. A museum is a place to learn about culture and history through exhibits. However, do museum visitors learn anything while they visit? If so, do museum visitors remember what they learned in the museums?
Many people who visit the museum simply vaguely look at the exhibits, or lose their direction due to a large number of exhibits and the amount of information related to them. Such a visit will not be remembered and will not be established as knowledge. Therefore, in this study, we used quizzes about exhibits to enhance the learning experiences of museum visitors. By solving quizzes about the exhibits during the museum visit, a visitor can think more actively and understand the exhibits more deeply.
Let us think of an example. Now, a museum visitor is looking at gongs used in Cambodian rituals. The visitor uses the electronic guide application to take a quiz about the gongs. The visitor needs to actively refer to the actual exhibits and their descriptions to look for hints of the quiz’s answer. By taking quiz on Cambodian gongs, the visitor may also develop an interest in the “rituals” and the “musical instruments”. This will lead to the appreciation of similar exhibits, such as the vertical flute used in Bulgarian rituals. Therefore, the visitor may also try to answer a quiz on the Bulgarian vertical flute. Based on the knowledge gained during the visit, the visitor can compare the exhibits and discover similarities between them, which can lead to a deeper understanding of the exhibits. This way of viewing the exhibits is considered to be a great learning experience for the visitor.
In this study, we focus on the National Museum of Ethnology, Japan (a.k.a. Minpaku). We implement the proposed method on the Minpaku’s existing electronic guide application, which we have developed before. A visitor can use the Minpaku’s electronic guide application and enjoy quizzes during the visit. An example quiz is shown in Figure 1.
In recent years, research on the use of digital device in museums is conducted. Klopfer et al.
[
In a museum visit, it is important to provide contents according to the visitor’s interests.
There have been studies on personalization of museum contents. Wang et al. [
This section describes the quiz generation method. First, we describe problem definition in 3.1 and the data was used in this study in 3.2. Approach to quiz generation and details of the system are described in 3.3 and 3.4, respectively.
In this study, we develop a system for generating quizzes. A quiz to be generated is a threechoice question. There quiz question is a sentence which is missing a word. The goal is to guess the correct word from three choices.
First, we explain the problem definition for the quiz generation system. The input and the output of the quiz generation system is as follows.
Output The given sentence which has one word removed and three choices to be filled in the removed part. One word is the correct choice (the removed word itself). The rest two words are the incorrect choices.
We will give a concrete example using an actual exhibit in the Minpaku. Suppose that a visitor is looking at a drum made by carving wood, which is used in the rituals of a country in the Oceania region (Figure 2). From the description of the exhibit, it contains the information “Used to send signal, such as during rituals”. This description has the information “ritual”. The generated quiz question and choices are as follows: • Used to send [ ? ], such as during rituals
In this study, we use descriptions of the exhibits to generate quizzes. Descriptions are extracted from the database provided by the Minpaku, which contains detailed information and images about the exhibits in the Minpaku. The database contains 72,428 items of data. Table 1 shows an example of data in the Minpaku’s database. The descriptions about an exhibit contains the following information: • Usage, • Fabrication method and materials, • Transition and distribution, • Other information.
In this study, we use a language model called BERT [7] to generate quizzes. Typically, a language model is used to predict the next word in a sentence using only previous words. BERT is able to predict a word in the middle of a sentence given words from both sides. For example, given the sentence “The [ ? ] of Japan is Tokyo”, BERT can fill the word “capital” into the blank position, using information of the rest words. Output of BERT is not only the best word, but also other candidates and their probabilities. We use the same approach to generate incorrect choices by using predictions of BERT on a quiz question that has a blank part.
We use the pre-trained BERT model for Japanese published by Inui and Suzuki Laboratory at Tohoku University. The model was pre-trained with data from Japanese Wikipedia.
In this study, we perform additional training on the above pre-trained BERT model using the descriptions of the exhibits from the database provided by Minpaku.
In addition, we also use Japanese WordNet [8] for quiz choices generation. WordNet is a thesaurus database systematized by hypernym or hyponym. We use WordNet as a filter to remove inappropriate candidate words outputted by BERT.
This section describes the quiz generation system. The quiz generation system consists of the two steps: quiz question generation and incorrect choices selection The process of the quiz generation system is described below with an example using the exhibit shown in Figure 2. 3.4.1. Quiz question generation The first step is to generate the quiz question. Quiz question is generated from the description of a exhibit by replacing one word from it with the notation “[ ? ]”. The word to be replaced is a noun, since nouns usually provide important information of a sentence. To select the replaced word, we firstly use the morphological analysis software MeCab [ 9] to split the description into parts of speech, get all the nouns and randomly select one word. Some nouns such as pronouns are set as stop words since the may not directly represent the content, and will not be selected. The quiz question is then generated by replacing the selected word in the description with the notation “[ ? ]”. The “[ ? ]” in the description indicates that the part is a blank. For example, consider the case where the description is “Used to send signal, such as during rituals”. The nouns in the description are: signal and ritual. We randomly select one word from these two. If “signal” is selected, the quiz question will be as follows:
The selected word, which is signal in this case, is the correct choice among the quiz choices. 3.4.2. Incorrect choices selection The second step is to select quiz’s incorrect choices. This step generates incorrect choices that properly fit into the blank in a quiz question. The BERT language model is used in this step. In this study, we use a BERT model which has been pre-trained using from Japanese Wikipedia. We also implemented additional training on the pre-trained model using descriptions of exhibits from Minpaku’s database.
Since BERT is able to give predictions of a blank part in a sentence using the rest words, we used BERT to output candidates for incorrect choices. Given a quiz question generated from the previous step which has a word replace by the notation “[ ?]”, we use BERT to output candidates to be filled in the “[ ? ]” and use them as incorrect choices of the quiz. In the phenomenon that some of the candidates are hypernyms or hyponyms of the correct choice, the quiz becomes inappropriate. To solve this problem, we use Japanese WordNet to find out hypernyms and hyponyms of the correct choice from candidates outputted by BERT. The incorrect choices of a quiz are selected so that there are no hypernyms and hyponyms of the correct choice.
We explain details of the process in this step using an example. Consider that quiz question and correct choice generated from the first step is as follows:
Correct choice: signal.
We use BERT to predict the “[ ? ]” part of the quiz question. The output candidates are:
Since the correct choice is signal, we use WordNet to find out hypernyms and hyponyms of signal from the above candidates, which are: sign and cue, and remove them from the list of candidates. After that, we randomly select two words from the remaining candidates and use them as quiz’s incorrect choices, which can be: First incorrect choice: message Second incorrect choice: letter
This section describes the user experiment conducted in this research and its results.
A user experiment has been conducted to evaluate the proposed quiz generation system. We prepared two group of subjects where the first group consists of seven university students and the second group consists of four university students. Subjects from both groups use the Minpaku’s electronic guide application during an actual two-hour visit to Minpaku. Each group took the visit with the following conditions:
After the visit, subjects from both groups were asked whether the experience of a visit to museum was improved or not using the electronic guide application. In this study, we used a questionnaire with 13 questions, where answers for each question is rated on a 5-point Likert scale where 1=not at all and 5=totally agree.
In particular, for the seventh question, which is “Could you understand the information about the exhibits?", the average answer was 2.71 for a visit without the quizzes compared to a value of 4.00 for a visit with the quizzes, which is a significant improvement. This indicates that the use of quizzes in the visit can deepen the understanding of the exhibits. In addition, the quizzes can be provided an opportunity for visitors to become interested in the exhibits. In this way, it was shown that the quiz was efective in improving the learning experience of the museum visit.
In this study, we proposed a method to enhance the learning experience of visitors to museums using quizzes about exhibits.
We developed a system that automatically generates quizzes from descriptions of exhibits using a fine-tuned BERT model. The system was implemented on the Minpaku’s electronic guide application. By solving quizzes about the exhibits during the museum visit, a visitor can think more actively and understand the exhibits more deeply.
We also conducted an user experiment to evaluate the developed system. Participants of the experiment used the Minpaku’s electronic guide application with the quiz system installed during an actual visit to Minpaku, and answered to several questions at the end of the visit. As a result, we confirm that resolving quizzes during a museum tour is efective in getting interested in the exhibits and understanding the information about the exhibits.
This work was supported in part by JSPS KAKENHI Grant Numbers JP18H03494, JP18H03243, JP18K18161, JP17H00762, and JP16H01756. It was also supported by the NII Open Collaborative Research 2020 Program, numbers 20S1001, 20AD03 and 20AD04.
We sincerely thank the Japanese National Museum of Ethnology (Minpaku) for ofering the exhibit meta-data. [5] E. Van Dijk, A. Lingnau, H. Kockelkorn, Measuring enjoyment of an interactive museum experience, in: Proceedings of the 14th ACM International Conference on Multimodal Interaction, 2012, pp. 249–256. [6] T. Kuflik, E. Minkov, K. Kahanov, Graph-based recommendation in the museum., in: Proceedings of the 1st International Workshop on Decision Making and Recommender Systems, 2014, pp. 46–48. [7] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, Journal of arXiv preprint arXiv:1810.04805 (2018). [8] H. Isahara, F. Bond, K. Uchimoto, M. Utiyama, K. Kanzaki, Development of the japanese wordnet, in: Proceedings of the 6th International Conference on Language Resources and Evaluation, 2008, pp. 2420–2423. [9] T. Kudo, K. Yamamoto, Y. Matsumoto, Applying conditional random fields to japanese morphological analysis, in: Proceedings of the 9th Conference on Empirical Methods in Natural Language Processing, 2004, pp. 230–237.