Location-based Reminder for Memorizing What Visitors Learn at a Museum Yoshiyuki Shoji1 , Kenro Aihara2,3 , Martin J. Dürst1 , Noriko Kando2,3 , Takuya Nakaya1 , Hiroaki Ohshima4 , Takehiro Yamamoto5 and Yusuke Yamamoto6 1 Aoyama Gakuin University, 5-10-1 Fuchinobe, Chuo-ku, Sagamihara, Kanagawa 252-5258, Japan 2 National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan 3 The Graduate University for Advanced Studies, 2-1-2 Hitotsubashi, Chiyoda-ku Tokyo, 101-8430, Japan 4 University of Hyogo, 7-1-28 Minatojima-minamimachi, Chuo-ku, Kobe, Hyogo, 650-0047, Japan 5 University of Hyogo, 8-2-1 Gakuennishi-machi, Nishi-ku, Kobe, Hyogo, 651-2197, Japan 6 Shizuoka University, 3-5-1 Johoku, Naka-ku, Hamamatsu, Shizuoka 432-8011, Japan Abstract This paper proposes a method to promote a strongly memorable museum experience by sending re- minder notifications when visitors approach places related to the exhibits. Memory retention studies have reported that memory that is linked to personal experience is not easily forgotten. Our method estimates real-world places that are related to a museum exhibit by linking characteristic terms that ap- pear in reviews for these places with exhibit commentaries. Relevance between the characteristic terms is calculated by their co-occurrence in documents and web search query suggestions. This enables us to send reminders about exhibits to users’ mobile devices when they approach related locations in their daily life. Through experiments conducted at the National Museum of Ethnology in Japan, participants who received notifications about the exhibits remembered more exhibits one week later. Keywords Museum, Memory Retention, Notification, Reminder 1. Introduction People easily forget what they have learned in books, classes, and museums. The content of this paper will also be forgotten; however, this paper proposes a technique for remembering it. As the saying “Example is better than precept” goes, people are more likely to remember information that is connected to their personal experiences, rather than the knowledge that is simply seen or heard. Human memory can be roughly divided into two types: semantic and episodic [1, 2]. Semantic memory consists of facts and knowledge which, while long term, WEPIR 2021: The Third Workshop on Evaluation of Personalisation in Information Retrieval at CHIIR 2021, 19 March 2021. Virtual event. " shoji@it.aoyama.ac.jp (Y. Shoji); kenro.aihara@nii.ac.jp (K. Aihara); duerst@it.aoyama.ac.jp (M. J. Dürst); kando@nii.ac.jp (N. Kando); nakaya@sw.it.aoyama.ac.jp (T. Nakaya); ohshima@ai.u-hyogo.ac.jp (H. Ohshima); t.yamamoto@sis.u-hyogo.ac.jp (T. Yamamoto); yamamoto@inf.shizuoka.ac.jp (Y. Yamamoto) ~ http://shoji-lab.jp/ (Y. Shoji)  0000-0002-7405-9270 (Y. Shoji); 0000-0002-4287-2523 (K. Aihara); 0000-0001-7568-0766 (M. J. Dürst); 0000-0002-2133-0215 (N. Kando); 0000-0002-9492-2246 (H. Ohshima); 0000-0003-0601-3139 (T. Yamamoto); 0000-0001-9829-6521 (Y. Yamamoto) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) 79 Name of Location “You seem to be in front of a certain osteopathic clinic” Name of Exhibit “Please think about Chikunza right here!” Snippet of Exhibit Description “This is because the description refers to a liturgical play of a dancer wearing the mask to treat boys” Snippet of Place Review (Omitted due to character limit) “And the review of this location says, I had my body taken care of here” Figure 1: Sample of notification when a user approaches an osteopathic clinic. are difficult to remember. Episodic memory consists of personal experiences which are not long-term on their own, but, because they are connected to personal experiences, are easier to remember and recall. Such experience-based episodes are likely to be recalled over and over again. Thus, an important episode can become a semantic memory and, finally, personal knowledge. To make what visitors see on-site at a museum or in a museum catalog more memorable, we propose a method that strongly connects exhibits to their personal experience. Museums are extraordinary and not connected to our personal lives. As a result, there is little opportunity to recall exhibits in our daily life. Our method focuses on geographical entities (e.g., landmarks, stores, restaurants) to connect exhibits with experiences in the real world. Imagine that a visitor visited a museum to learn about ethnology. He or she was interested in a morin khuur (a Mongolian fiddle) in the museum. However, the knowledge of “what is a morin khuur and what are its characteristics” does not make concrete sense and, after a while, maybe forgotten. Therefore, when a visitor nears an apparel store, a notification is sent to his or her smartphone that says, “You are now visiting the shop of an apparel brand that is famous for having a horse as its logo. The horse is actually closely related to the morin khuur! The morin khuur is shaped like a horse’s head, and there is a founding myth about it.” By receiving such a notification, the existence of the morin khuur is connected to the individual’s experience and, therefore, this visitor will be more likely to remember this experience. In the future, when he or she passes by this apparel store, they may recall and think about the morin khuur. In this way, 80 the location-based reminder notification makes the museum visit and exhibit more memorable. This kind of attempt to fix textbook knowledge into memory by linking knowledge to places or experiences is already common in education. One of the most typical examples is elementary school field trips; students walk around a city with a teacher and are introduced to objects related to what they are learning in school. For example, the meaning of foreign terms on a signboard can be linked to language learning and the color of hydrangea flowers can be linked to pH (power of hydrogen) in science experiments. These methods are in accordance with the Meaningful Reception Learning theory [3]. Our method analyzes the user behavior log of a museum guide application and estimates the exhibits that the visitor is interested in from the log. For each of these exhibits, the method extracts keywords that characterize the exhibit from its description. It also collects review texts from review sites for all places in a specified area. The relevance between the terms in the exhibit’s description and the reviews are calculated, and a notification is created. Visitors can receive notification messages on a mobile device (e.g., smartphone), such as “You are in an apparel store right now. Actually, this store is associated with the morin khuur because the morin khuur is shaped like a horse’s head and a review of the store mentions a horse’s head.” Figure 1 shows an actual example of our system in which a notification about a mask was sent to users who approached an osteopathic clinic. This mask (called a Chikunza) is worn by a dancer of an African tribe who is in charge of supporting boys in a liturgical play. This is because terms such as “dancer and treat” are associated with the terms “body and care” in the osteopath’s review. One of the aims of this research is to propose an idea that evaluates the quality of information access in museums by measuring how much of the visiting experience is retained in the visitor’s knowledge. For this purpose, we created an application that actually aids memory retention. To verify the effectiveness of our method, we conducted an experiment at the Museum of Ethnology in Japan (called “Minpaku”). We used the operation log of the “Minpaku Guide”: an iPad guide app for the museum originally developed by us. Participants received notifications and were asked whether they remembered the exhibits they saw at the museum one week later. 2. Related Work This research is strongly related to studies on memory retention and museum experience, and uses the technology of geographical information recommendations and related term extraction. Research on memory in the field of psychology has been actively conducted for a long time. Bower et al. [4] proposed that people remember things longer when they associate them with their own memories. Similarly, Weinberg et al. [5] showed that when people move their bodies during an experience, it is more likely to be remembered. The traditional mnemonics method of loci [6] also supports the usefulness of linking a place and a topic. Memory retention has also been studied in the field of informatics. For example, Qiu et al. [7] discuss how user interaction in information retrieval affects the memorability of the search results. Our research also aims to make the results of information access in museums and guide apps more memorable through physical experience. There has already been some research on making museum experiences more memorable. 81 Falk et al. [8] state that to make museum experiences knowledgeable, it is important to connect the visiting experience with personal experience and to incorporate it into personal contexts. Petrelli et al. [9] are conducting research on encouraging people to look back on their museum experience by providing personalized online souvenirs. The recommendation of places based on the user’s interests and preferences has also been widely studied. Koren et al. [10] proposed an audio guide that provides personalized recom- mendations of locations based on the user’s interests. To link the learning content to geographic entities, the method must calculate the relevance between two keywords. As a concept-based method, Watabe et al. [11] calculated the relevance between two terms from the co-occurrence in the concept space. Shirakawa et al. [12] uses Wikipedia; they proposed a method to express the relationship between two terms using the Wikipedia Thesaurus and combining it with Web searches. In this study, our method follows these methods to link the terms in the descriptions of exhibits with those in a place’s review. 3. Method This section describes our method of linking exhibits and geographical entities. The method consists of three steps: 1. extracting characteristic terms from the text of the exhibit’s commentary and the place’s review, 2. calculating the relevancy between characteristic terms, and 3. sending reminder notifications to a smart device. First, our method extracts characteristic terms from the exhibits and place reviews. We used term frequency-inverse document frequency (TF-IDF) for this extraction. Sentences were divided into terms by a morphological analyzer (terms in Japanese sentences are not set off by spaces), and verbs and nouns were extracted. The frequency with which a term appeared in all the descriptions and in all the reviews was counted, and the terms that characterize certain exhibits and places were ranked by the TF-IDF score. The method then calculates the relevance of a combination of characteristic terms of exhibits and characteristic terms of reviews for places. We calculated the relevance based on three hypotheses: co-occurrence, category distance, and query suggestion. The co-occurrence score is based on the hypothesis that related terms are more likely to appear in the same document. For this purpose, we calculated the percentage of terms that co-occur in all Wikipedia articles. The category distance score is based on the hypothesis that terms that are semantically close to each other are also close in taxonomy. The method uses category information taken from Wikipedia. The score of a term contained in the title of a Wikipedia article was calculated based on the degree of overlap of the category to which these articles belong. The query suggestion score is intended to be slightly different from the other relevance scores; it takes into account the amusingness and popularity of the combination. To do this, we hypothesized that popular combinations of terms would appear in the users’ search queries. Therefore, the method sets the score as 1 if the suggested query of the search engine for the term contained another term; otherwise, the score is set as 0. 82 During the actual calculation, these three values are weighted and summed up. For the experiment, the co-occurrence weight was set to 0.2 and the category distance weight was set to 0.8. However, if a term was included in the query suggestion, the final score was unconditionally fixed at 1.0. These weights and conditions were determined empirically through preliminary experiments. It is rare for a term to accidentally appear at the top of a query suggestion and, when it does, the combination is meaningful. Finally, the method actually links the place to the exhibit by aggregating the degree of relevance between their characteristic terms. In this experiment, places were targeted for notification if even one term with a high degree of relevance was included in the review. Each exhibit has a ranking of which sentences in the review it is deeply related to. For the top 10 exhibits that the visitor was interested in, the system decides in advance which sentences will be displayed when the user approaches the location. Every review is linked to a real place and these places have location information (i.e., latitude and longitude). Using the GPS information of the user’s mobile device, the geofence function of the smartphone displays a notification when the user comes within a 100-meter radius of the target location. In the application for this experiment, we used the local notification function without accessing a server because we were able to pre-calculate where to send the notification in advance. As shown in Fig. 1, the title of the actual notification message includes the name of the place, the name of the exhibit, and an image of the exhibit. The aim is to assist memorization by using paired-association-like learning, such as when someone says “clinic”, you say “Chikunza”. The reason why this place was recommended is presented under the title. Single sentences containing the relevant characteristic term are extracted from the exhibit description and the place review text, and each is displayed as a snippet. The notification text is written in a catchy, broken tone. Tapping on the notification displays the full text of the omitted snippets and a higher resolution image. The user can select one of two buttons, “Aha” or “Not interested”, and they can also choose to ignore the notification. 4. Experiment We conducted two experiments to evaluate whether reminders affect visitor memory retention. The first was a detailed comparison to clarify whether notifications help users retain textbook- like knowledge. Participants did not go to the museum site, but rather used the app as a catalog in the lab. The other was a field user experiment that evaluated the degree of memory retention and operational problems when such a system was actually used at the museum site. In both of these experiments, participants went out in the city and received notifications (except participants in the control groups), and one week later were asked and answered how many of their favorite exhibits they remembered. 4.1. Dataset For the experiments, we used the exhibit data of the National Museum of Ethnology (Minpaku), Japan. We used Minpaku Guide; an iPad app originally developed by us for navigating in the museum. Participants can easily search and browse detailed information on 3,031 exhibits that 83 are sampled from over 10,000 exhibits at the museum. This app can be used as a guide on-site at the museum or as a catalog at home. The app collects user browsing logs and explicitly lists their favorite exhibits. These features were used to determine which exhibits to use for reminders. As place data, we collected online reviews for locations from Google Maps: 7,393 places in a city in the western Tokyo area (i.e., near to the first author’s laboratory) and 18,165 places in a city in the Kobe area (i.e., near to the museum). Both areas are two kilometers square and include the downtown area around a terminal station. All places in the dataset have one or more text reviews, not just star ratings. 4.2. Comparison Experiment at Laboratory To evaluate whether notification is effective for knowledge retention, we conducted a subject experiment. We divided eight participants into three groups: • in City: Four participants walked through the city and received notifications, • in Lab: Two participants stayed in the laboratory and received notifications while indoors, and • no Reminder: Two participants received no notifications at all. All participants operated the guide application freely for one hour in the laboratory. After that, the top 10 exhibits with the highest number of accesses were extracted from the logs. We lent iPhones to two groups of participants, in City and in Lab. The in City participants went to the downtown area in front of a station and walked through the area for 60 minutes. During this stroll, they were sent notifications when they approached places related to their favorite exhibits. The in Lab participants spent their free time indoors. Regardless of the participant’s behavior or current location, the participants received notifications at random times. The no Reminder participants used the catalog app and then went home without doing anything. One week later, we checked how many of the top 10 items the participants had viewed were remembered. We performed this check in two stages. The first step was to answer the question without hints; the participants responded by listing all the exhibits they remembered. The second was with hints; the participants were first asked to “recall the location where you received the notification.” After that, the participants provided additional exhibits that they could recall by associating them with the location. The results of the experiment are shown on the left side of Table 1. The participants in the in Lab group remembered the most exhibits and there were exhibits that they were able to recall with hints. 4.3. User Case Study Onsite at Museum As a more realistic problem setting, we conducted an additional experiment that sent notifica- tions while participants were walking around the city after an onsite visit to the museum. Five participants visited the National Museum of Ethnology and appreciated its exhibits for about two hours. They used the Minpaku Guide and selected 15 exhibits by using the app’s favorite list function. Ten of their favorite exhibits were targeted for notification. A week later, while 84 Table 1 Results of the two memory experiments, including the average number of exhibits for each condition, and the ratio of exhibits remembered and received notifications. Comparison at Laboratory Museum in City in Lab no Reminder in City Number of notifications 8.25 7.00 - 9.80 Number of exhibits remembered without hint 5.75 5.00 5.50 5.80 Number of exhibits remembered with hint 0.25 2.50 - 0.20 Total number of exhibits remembered 6.00 7.50 5.50 6.00 Ratio of remember / notification 0.57 0.50 - 0.61 walking around the city for another event, our system sent reminders of the exhibits to their iPhones. After an additional week, we asked what exhibits they remembered. The aim of this experiment was to disclose problems with and improvements that can be made to the system when it is operated in the field. For this purpose, we conducted a detailed questionnaire on the system’s usability. The results of the experiment are shown on the right side of Table 1. The ratio of the exhibits that were remembered of the 10 exhibits for which notifications were sent was 0.61 and 0.31 for the five exhibits without notification. Two weeks had passed since the visit, so the results cannot be directly compared with the results of the previous experiment; however, most participants remembered half of the exhibits. Similar to the comparison experiment, providing hints proved effective for recall. As to usability, we obtained opinions such as “it is annoying when multiple exhibits are notified in the same place” and “when notifications are displayed in rapid succession, the previous notification became inaccessible.” 5. Discussion Overall, reminding by notification seemed effective for memory retention. In the comparison experiment, the number of exhibits remembered by the participants differed greatly depending on whether they received a notification or not. On the other hand, we could not confirm the effectiveness of linking the notification to places. The participants who received notifications while indoors remembered the most exhibits. This may be due to the experimental setting; the participants in the in City group were forced to walk around the city, which is more unusual than remaining in an indoor setting. These participants may have been so busy or distracted while walking around the city that they did not read the notifications carefully. In the case of recall with hints, the participants commented that inappropriate or unexpected combinations helped their recall. The degree of relevance between characteristic terms is important, but the unexpectedness of the relationship between exhibits and place may also be influential. The participants pointed out cases where the quality of the review text was low and unrelated to the place. Prior cleansing is therefore needed. Regarding the user case study, the frequency of notifications is an issue that needs to be resolved. In this experiment, the notifications were not grouped by location, but were handled in 85 units of sentences. As a result, several exhibits were linked to one place and several notifications were issued in close proximity. In actual operation, it is necessary to select the target place so that notifications are sent out at equal intervals throughout the city. 6. Conclusion In this research, we proposed a method to help memory retention by linking real places to museum exhibits and sending out notification reminders. The experimental results showed that the reminders increased the number of exhibits remembered by the participants one week later. On the other hand, we were not able to prove the effect of linking it to the experience of going out into the city. As a future challenge, larger-scale and more sophisticated experiments are needed. We should conduct comparative experiments at a museum site and in a more natural environment to send notifications. Currently, the participants are forced to walk around the city in the experiment setting. This situation is unnatural and cannot connect the exhibits to their real, daily lives. Personalization is also an important issue. There was a bias in the areas of interest among the participants and this affected memory retention. The calculation of the relationship between local objects and exhibits also needs to be made more precise. In closing, please say aloud “I read a paper about sending a notification of an exhibit at the station!” the next time you go to the station. This will convert the content of this paper into an experiential memory for you so that it will not be easily forgotten. Acknowledgments This work was supported in part by JSPS KAKENHI Grant Numbers JP18H03494, JP18H03243, JP18K18161, JP17H00762, and JP16H01756. It was also supported by the NII Open Collaborative Research 2020 Program, numbers 20S1001, 20AD03, and 20AD04. We sincerely thank the Japanese National Museum of Ethnology (Minpaku) for offering the exhibit meta-data. References [1] E. Tulving, Elements of episodic memory, Oxford Psychology Series (1983). [2] E. Tulving, What is episodic memory?, Current directions in psychological science 2 (1993) 67 – 70. [3] D. P. Ausubel, Meaningful reception learning and the acquisition of concepts, in: Analyses of concept learning, Elsevier, 1966, pp. 157 – 175. [4] G. H. Bower, S. G. Gilligan, Remembering information related to one’s self, Journal of Research in Personality 13 (1979) 420 – 432. [5] L. Weinberg, A. Hasni, M. Shinohara, A. Duarte, A single bout of resistance exercise can enhance episodic memory performance, Acta Psychologica 153 (2014) 13 – 19. [6] F. A. Yates, The art of memory, volume 64, Random House, 1992. 86 [7] S. Qiu, U. Gadiraju, A. Bozzon, Towards memorable information retrieval, in: Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval, 2020, pp. 69 – 76. [8] J. H. Falk, L. D. Dierking, Learning from museums, Rowman & Littlefield, 2018. [9] D. Petrelli, M. T. Marshall, S. O’Brien, P. McEntaggart, I. Gwilt, Tangible data souvenirs as a bridge between a physical museum visit and online digital experience, Personal and Ubiquitous Computing 21 (2017) 281 – 295. [10] A. Koren, N. Stash, A. Andreev, A proposal for semantic recommender for outdoor audio tour guides, in: Workshop on Personalization in Mobile Applications at the 5th ACM Conference on Recommender Systems. Chicago, IL, USA, 2011, pp. 1 – 4. [11] H. Watabe, N. Okumura, T. Kawaoka, The method of measuring the degree of association between concepts using attributes of the concepts and coincidence information, Journal of Natural Language Processing 13 (2006) 53 – 74. [12] M. Shirakawa, K. Nakayama, E. Aramaki, T. Hara, S. Nishio, Relation extraction between related concepts by combining wikipedia and web information for japanese language, in: Information Retrieval Technology, Springer, 2010, pp. 310 – 319. 87