Quiz Generation on the Electronic Guide Application for Improving Learning Experience in the Museum Masaki Ueta1 , Tomoya Hashiguchi1 , Huu-Long Pham1 , Yoshiyuki Shoji2 , Noriko Kando3,4 , Yusuke Yamamoto5 , Takehiro Yamamoto6 and Hiroaki Ohshima1 1 University of Hyogo, 7-1-28 Minatojima-minamimachi, Chuo-ku, Kobe, Hyogo 650-0047, Japan 2 Aoyama Gakuin University, 5-10-1 Fuchinobe, Chuo-ku, Sagamihara, Kanagawa 252-5258, Japan 3 National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan 4 The Graduate University for Advanced Studies, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan 5 Shizuoka University, 3-5-1 Johoku, Naka-ku, Hamamatsu, Shizuoka 432-8011, Japan 6 University of Hyogo, 8-2-1 Gakuennishi-machi, Nishi-ku, Kobe, Hyogo 651-2197, Japan Abstract We propose a method to generate quizzes on a museum electronic guide application. While a museum is considered to be a place for learning, it would be hard for a visitor to actively appreciate the exhibits in the museum, especially when they have little knowledge about the exhibits. In this study, we develop a method that automatically generates quizzes about the exhibits on an electronic guide application. The proposed method utilizes a BERT model that is trained with the additional corpus constructed from the descriptions of the exhibits and automatically generates a quiz about exhibit. By solving quizzes about the exhibits during the museum visit, we expect that a visitor’s museum experience would be more active, and they would understand the exhibits more deeply. We implement the proposed method on the electronic guide application that is designed for the National Museum of Ethnology, Japan (a.k.a. Minpaku). Keywords Museum visit, Quiz generation, Personalization 1. Introduction There are many museums around the world. A museum is a place to learn about culture and history through exhibits. However, do museum visitors learn anything while they visit? If so, do museum visitors remember what they learned in the museums? Many people who visit the museum simply vaguely look at the exhibits, or lose their direction due to a large number of exhibits and the amount of information related to them. Such a visit will not be remembered and will not be established as knowledge. Therefore, in this study, WEPIR 2021: The Third Workshop on Evaluation of Personalisation in Information Retrieval at CHIIR 2021, 19 March 2021. Virtual event. " aa20t502@ai.u-hyogo.ac.jp (M. Ueta); aa19j508@ai.u-hyogo.ac.jp (T. Hashiguchi); aa19e510@ai.u-hyogo.ac.jp (H. Pham); shoji@it.aoyama.ac.jp (Y. Shoji); kando@nii.ac.jp (N. Kando); yamamoto@inf.shizuoka.ac.jp (Y. Yamamoto); t.yamamoto@sis.u-hyogo.ac.jp (T. Yamamoto); ohshima@ai.u-hyogo.ac.jp (H. Ohshima)  0000-0002-7405-9270 (Y. Shoji); 0000-0002-2133-0215 (N. Kando); 0000-0001-9829-6521 (Y. Yamamoto); 0000-0003-0601-3139 (T. Yamamoto); 0000-0002-9492-2246 (H. Ohshima) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 96 Figure 1: An example quiz on the Minpaku’s electronic guide application (originally in Japanese) we used quizzes about exhibits to enhance the learning experiences of museum visitors. By solving quizzes about the exhibits during the museum visit, a visitor can think more actively and understand the exhibits more deeply. Let us think of an example. Now, a museum visitor is looking at gongs used in Cambodian rituals. The visitor uses the electronic guide application to take a quiz about the gongs. The visitor needs to actively refer to the actual exhibits and their descriptions to look for hints of the quiz’s answer. By taking quiz on Cambodian gongs, the visitor may also develop an interest in the “rituals” and the “musical instruments”. This will lead to the appreciation of similar exhibits, such as the vertical flute used in Bulgarian rituals. Therefore, the visitor may also try to answer a quiz on the Bulgarian vertical flute. Based on the knowledge gained during the visit, the visitor can compare the exhibits and discover similarities between them, which can lead to a deeper understanding of the exhibits. This way of viewing the exhibits is considered to be a great learning experience for the visitor. In this study, we focus on the National Museum of Ethnology, Japan (a.k.a. Minpaku). We implement the proposed method on the Minpaku’s existing electronic guide application, which we have developed before. A visitor can use the Minpaku’s electronic guide application and enjoy quizzes during the visit. An example quiz is shown in Figure 1. 97 2. Related Work In recent years, research on the use of digital device in museums is conducted. Klopfer et al. [1] conducted a riddle solving game using digital device for museum education. The purpose of this study is to encourage visitors to refer to the exhibits and communication among visitors. As a result, visitors were actively looking for and referring to the exhibits that provided the answers. It was also suggested that the problem of visitors focusing only on digital devices could be solved by incorporating a riddle solving game. Robert et al. [2] showed that by using quizzes in museums, visitors actively referred to both the exhibit descriptions and the electronic guides. In a museum visit, it is important to provide contents according to the visitor’s interests. There have been studies on personalization of museum contents. Wang et al. [3] [4] thought that the information should be based on the visitor’s own interests and context. Based on this idea, they developed a personalized museum visit program using the visitor’s interest and context information. Dijk et al. [5] used game-style questions about the topic of the exhibit at the beginning of the museum visit. Using the answers to the questions, they showed visitors a personalized visit route. Kuflik et al. [6] obtained information about visitors’ needs and interests. Using this information, they developed a graph-based recommendation system that recommends relevant information from the museum’s own information. 3. Quiz generation method This section describes the quiz generation method. First, we describe problem definition in 3.1 and the data was used in this study in 3.2. Approach to quiz generation and details of the system are described in 3.3 and 3.4, respectively. 3.1. Problem Definition In this study, we develop a system for generating quizzes. A quiz to be generated is a three- choice question. There quiz question is a sentence which is missing a word. The goal is to guess the correct word from three choices. First, we explain the problem definition for the quiz generation system. The input and the output of the quiz generation system is as follows. Input A sentence. Output The given sentence which has one word removed and three choices to be filled in the removed part. One word is the correct choice (the removed word itself). The rest two words are the incorrect choices. We will give a concrete example using an actual exhibit in the Minpaku. Suppose that a visitor is looking at a drum made by carving wood, which is used in the rituals of a country in the Oceania region (Figure 2). From the description of the exhibit, it contains the information “Used to send signal, such as during rituals”. This description has the information “ritual”. The generated quiz question and choices are as follows: 98 Figure 2: Visitors admiring the exhibits • Used to send [ ? ], such as during rituals 1. signal 2. message 3. letter Of the choices, “signal” is the correct choice. 3.2. Database of the National Museum of Ethnology In this study, we use descriptions of the exhibits to generate quizzes. Descriptions are extracted from the database provided by the Minpaku, which contains detailed information and images about the exhibits in the Minpaku. The database contains 72,428 items of data. Table 1 shows an example of data in the Minpaku’s database. The descriptions about an exhibit contains the following information: • Usage, • Fabrication method and materials, • Transition and distribution, • Other information. We use the above information for quiz generation. 99 Table 1 An example of data in the Minpaku’s database Exhibit ID K0006979 Exhibit name Splintered wood drum User Specific adult men Location New Hebrides islands, Ambrym island, Ranon village Collection date 1969-01-14 Usage and how to use Music instrument. Used to send signal, such as during rituals. Beat the body with a wooden stick. Fabrication method and materials Fabrication method:Carving Material:wood 3.3. Approach to Quiz Generation In this study, we use a language model called BERT [7] to generate quizzes. Typically, a language model is used to predict the next word in a sentence using only previous words. BERT is able to predict a word in the middle of a sentence given words from both sides. For example, given the sentence “The [ ? ] of Japan is Tokyo”, BERT can fill the word “capital” into the blank position, using information of the rest words. Output of BERT is not only the best word, but also other candidates and their probabilities. We use the same approach to generate incorrect choices by using predictions of BERT on a quiz question that has a blank part. We use the pre-trained BERT model for Japanese published by Inui and Suzuki Laboratory at Tohoku University. The model was pre-trained with data from Japanese Wikipedia. In this study, we perform additional training on the above pre-trained BERT model using the descriptions of the exhibits from the database provided by Minpaku. In addition, we also use Japanese WordNet [8] for quiz choices generation. WordNet is a thesaurus database systematized by hypernym or hyponym. We use WordNet as a filter to remove inappropriate candidate words outputted by BERT. 3.4. Quiz Generation System This section describes the quiz generation system. The quiz generation system consists of the two steps: quiz question generation and incorrect choices selection The process of the quiz generation system is described below with an example using the exhibit shown in Figure 2. 3.4.1. Quiz question generation The first step is to generate the quiz question. Quiz question is generated from the description of a exhibit by replacing one word from it with the notation “[ ? ]”. The word to be replaced is a noun, since nouns usually provide important information of a sentence. To select the replaced word, we firstly use the morphological analysis software MeCab [9] to split the description into parts of speech, get all the nouns and randomly select one word. Some nouns such as pronouns are set as stop words since the may not directly represent the content, and will not be selected. The quiz question is then generated by replacing the selected word in the description with the 100 notation “[ ? ]”. The “[ ? ]” in the description indicates that the part is a blank. For example, consider the case where the description is “Used to send signal, such as during rituals”. The nouns in the description are: signal and ritual. We randomly select one word from these two. If “signal” is selected, the quiz question will be as follows: Quiz question: Used to send [ ? ], such as during rituals. The selected word, which is signal in this case, is the correct choice among the quiz choices. 3.4.2. Incorrect choices selection The second step is to select quiz’s incorrect choices. This step generates incorrect choices that properly fit into the blank in a quiz question. The BERT language model is used in this step. In this study, we use a BERT model which has been pre-trained using from Japanese Wikipedia. We also implemented additional training on the pre-trained model using descriptions of exhibits from Minpaku’s database. Since BERT is able to give predictions of a blank part in a sentence using the rest words, we used BERT to output candidates for incorrect choices. Given a quiz question generated from the previous step which has a word replace by the notation “[ ?]”, we use BERT to output candidates to be filled in the “[ ? ]” and use them as incorrect choices of the quiz. In the phenomenon that some of the candidates are hypernyms or hyponyms of the correct choice, the quiz becomes inappropriate. To solve this problem, we use Japanese WordNet to find out hypernyms and hyponyms of the correct choice from candidates outputted by BERT. The incorrect choices of a quiz are selected so that there are no hypernyms and hyponyms of the correct choice. We explain details of the process in this step using an example. Consider that quiz question and correct choice generated from the first step is as follows: Quiz question: Used to send [ ? ], such as during rituals. Correct choice: signal. We use BERT to predict the “[ ? ]” part of the quiz question. The output candidates are: Candidates: gift, food, sign, sentence, letter, thing, article, cue Since the correct choice is signal, we use WordNet to find out hypernyms and hyponyms of signal from the above candidates, which are: sign and cue, and remove them from the list of candidates. After that, we randomly select two words from the remaining candidates and use them as quiz’s incorrect choices, which can be: First incorrect choice: message Second incorrect choice: letter 4. User experiment and discussion This section describes the user experiment conducted in this research and its results. 101 Table 2 Questionnaires after observing the user experiment Question Without With questionnaire No. quiz system quiz system 1 Do you like visiting museums. 3.57 3.75 2 Did you have a certain idea of what you wanted to see? 2.00 3.00 Do you have memorable exhibits 3 2.00 3.25 from past museum visits? 4 Was the Minpaku’s guide application helpful? 3.14 4.00 5 Was the Minpaku’s guide application easy to use? 2.29 3.25 6 Were you able to develop an interest in the exhibits? 3.57 4.75 Could you understand the information 7 2.71 4.00 about the exhibits? 8 Did your willingness to look at exhibits increase? 3.86 4.50 Do you feel that you have gained knowledge 9 3.00 3.50 about the exhibits? 10 Did you satisfy with the visit in this time. 4.14 4.00 11 How fun was the visit. 4.43 4.25 12 Were you able to focus on viewing the exhibition. 3.43 4.00 13 Do you want to come to the Minpaku again? 3.86 3.50 4.1. Experimental Setup A user experiment has been conducted to evaluate the proposed quiz generation system. We prepared two group of subjects where the first group consists of seven university students and the second group consists of four university students. Subjects from both groups use the Minpaku’s electronic guide application during an actual two-hour visit to Minpaku. Each group took the visit with the following conditions: Group 1: Use the electronic guide application without quiz system. Group 2: Use the electronic guide application with quiz system. After the visit, subjects from both groups were asked whether the experience of a visit to museum was improved or not using the electronic guide application. In this study, we used a questionnaire with 13 questions, where answers for each question is rated on a 5-point Likert scale where 1=not at all and 5=totally agree. 4.2. Experimental Results and Discussion Table 2 shows the average responses to the questionnaire of each group, with quiz system or without quiz system. Among the questions in the questionnaire, we focus on question 6, 7, 8 and 9. Basically, the average values of the responses for the visit using the application with the quizzes is higher than without the quizzes. 102 In particular, for the seventh question, which is “Could you understand the information about the exhibits?", the average answer was 2.71 for a visit without the quizzes compared to a value of 4.00 for a visit with the quizzes, which is a significant improvement. This indicates that the use of quizzes in the visit can deepen the understanding of the exhibits. In addition, the quizzes can be provided an opportunity for visitors to become interested in the exhibits. In this way, it was shown that the quiz was effective in improving the learning experience of the museum visit. 5. Conclusion In this study, we proposed a method to enhance the learning experience of visitors to museums using quizzes about exhibits. We developed a system that automatically generates quizzes from descriptions of exhibits using a fine-tuned BERT model. The system was implemented on the Minpaku’s electronic guide application. By solving quizzes about the exhibits during the museum visit, a visitor can think more actively and understand the exhibits more deeply. We also conducted an user experiment to evaluate the developed system. Participants of the experiment used the Minpaku’s electronic guide application with the quiz system installed during an actual visit to Minpaku, and answered to several questions at the end of the visit. As a result, we confirm that resolving quizzes during a museum tour is effective in getting interested in the exhibits and understanding the information about the exhibits. Acknowledgments This work was supported in part by JSPS KAKENHI Grant Numbers JP18H03494, JP18H03243, JP18K18161, JP17H00762, and JP16H01756. It was also supported by the NII Open Collaborative Research 2020 Program, numbers 20S1001, 20AD03 and 20AD04. We sincerely thank the Japanese National Museum of Ethnology (Minpaku) for offering the exhibit meta-data. References [1] E. Klopfer, J. Perry, K. Squire, M.-F. Jan, C. Steinkuehler, Mystery at the museum a collabo- rative game for museum education, in: Proceedings of the 7th International Conference on Computer Support for Collaborative Learning, 2005, pp. 316–320. [2] J. Roberts, A. Banerjee, A. Hong, S. McGee, M. Horn, M. Matcuk, Digital exhibit labels in museums: Promoting visitor engagement with cultural artifacts, in: Proceedings of the 36th CHI Conference on Human Factors in Computing Systems, 2018, pp. 1–12. [3] Y. Wang, N. Stash, L. Aroyo, P. Gorgels, L. Rutledge, G. Schreiber, Recommendations based on semantically enriched museum collections, Journal of Web Semantics 6 (2008) 283–290. [4] Y. Wang, N. Stash, R. Sambeek, Y. Schuurmans, L. Aroyo, G. Schreiber, P. Gorgels, Cultivating personalized museum tours online and on-site, Journal of Interdisciplinary Science Reviews 34 (2009) 139–153. 103 [5] E. Van Dijk, A. Lingnau, H. Kockelkorn, Measuring enjoyment of an interactive museum experience, in: Proceedings of the 14th ACM International Conference on Multimodal Interaction, 2012, pp. 249–256. [6] T. Kuflik, E. Minkov, K. Kahanov, Graph-based recommendation in the museum., in: Proceedings of the 1st International Workshop on Decision Making and Recommender Systems, 2014, pp. 46–48. [7] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, Journal of arXiv preprint arXiv:1810.04805 (2018). [8] H. Isahara, F. Bond, K. Uchimoto, M. Utiyama, K. Kanzaki, Development of the japanese wordnet, in: Proceedings of the 6th International Conference on Language Resources and Evaluation, 2008, pp. 2420–2423. [9] T. Kudo, K. Yamamoto, Y. Matsumoto, Applying conditional random fields to japanese morphological analysis, in: Proceedings of the 9th Conference on Empirical Methods in Natural Language Processing, 2004, pp. 230–237. 104