Smart Museum Information Services to Assist Preservation, Transmission and Research in Cultural and Historical Heritage Domain Dmitry Korzun1 , Svetlana Yalovitsyna2 , and Valentina Volokhova3 1 Department of Computer Science, Petrozavodsk State University, Petrozavodsk, Russia dkorzun@cs.karelia.ru 2 Institute of Linguistics, Literature and History, Karelian Research Centre of Russian Academy of Science, Petrozavodsk, Russia jalov@yandex.ru 3 Department of Russian History, Petrozavodsk State University, Petrozavodsk, Russia vavolokhova@yandex.ru Abstract. Museums are now digitally enhanced based on smart infor- mation services that assist visitors and personnel. Cultural and historical heritage (CHH) is perceived in a personalized and cognitive way and new knowledge is created by the users. In this paper, we overview the smart museum concept. We consider services for effective CHH preservation and transmission within a smart museum. The services are extended to assist in research by discovering information for applying human exper- tise and reasoning knowledge. We introduce information representation models as a semantic network where data mining is reduced to ranking in the semantic network. As a case study we discuss the services for the History Museum of Petrozavodsk State University. Keywords: Cultural and historical heritage, Smart museum, Informa- tion services, Digital assistance, Semantic network, Data mining 1 Introduction Our society has entered into the digital era, and the term “smart” becomes widespread in relation to modern technologies [3]. We limit our scope with smart information and communication technology (ICT), which has multi-disciplinary character, and its “smart” aspects are now emerging in many domains, including social and cultural ones [7]. Cultural and historical heritage (CHH) is a domain where new ICT and digital services have a special impact on people approach to preservation, transmission, and research [5,6]. CHH entities can be considered in terms of various properties; to name some examples: works of fine and applied arts or folk crafts, archaeological, architectural, ethnographic or historical sites Lupeikiene A., Matulevičius R., Vasilecas O. (eds.): Baltic DB&IS 2018 Joint Proceedings of the Conference Forum and Doctoral Consortium. Copyright © 2018 for this paper by the papers' authors. Copying permitted for private and academic purposes. 2 and complexes, samples of park art and landscape architecture, industrial, docu- mental or audio-visual heritage, spoken tradition and language or literary values, customs, rituals, celebrations and beliefs, music, songs and dances, culinary and ethnological traditions, folk games and sports. In this paper, we consider the smart museum concept based on our previ- ous work [9,8,13]. We consider smart museum information services that can be developed based on a semantic network interlinking the museum CHH collec- tion, including knowledge acquired from visitors and museum personnel. The semantic network enhances the existing collection operating with digital repre- sentations of exhibits, descriptions of CHH-valued objects and facts as well as with any available fragments of CHH knowledge. This network is subject to data mining needed for selection of appropriate information as a result provided by services. Our case study is the History Museum of Petrozavodsk State Univer- sity (PetrSU) in respect to everyday life history; the museum provides the pilot testbed to analyze the information services. The rest of the paper is organized as follows. Section 2 overviews the recent progress in smart museum development. Section 3 introduces our smart mu- seum concept based on information services. Section 4 describes mathematical methods that our appraoch uses for services development. Section 5 discusses particular smart museum information services using the case study of the History Museum of PetrSU. Finally, Section 6 concludes the paper. 2 Smart Museum Development A lot of efforts have been made already in museum digitalization. Museum databases store the information part of the collection to track all knowledge related to and about the CHH objects (exhibits) and to ensure the long-term safety and sustainability of those objects within the museum’s care. The basic function is an electronic archive (catalogue) administered by museum personnel. Its extensions lead to “smart services”, emphasizing a certain intelligence level in service construction and delivery. The recent ICT progress (including Internet of Things, IoT) supports de- velopment of on-site personalized services for museum visitors. A visitor has a personal mobile device (e.g., smartphone) accessing relevant information about surrounding exhibits and in a personalized and cognitive way [2]. The infor- mation flow goes from digital cultural heritage to visitors. SMARTMUSEUM system [14] provides explanatory description and multimedia content associated with individual objects. A museum exhibition can tell a story [18]. Objects for a visitor to study can be recommended based on the user profile and context information. Additional content about an object can be retrieved from the Web. The on-site visit boundaries of CHH experience at the museum can be extended to assist the visitors during pre-visit planning and to follow up with post visit memories and reflections [10]. Museum information services support people to be involved into the process, e.g., using feedback when visitors leave posts on exhibits (to read by other vis- 44 3 itors) or evaluate exhibits (collaborative activity). IoT-enabled location-based services make possible shortening the information distance between objects in cultural spaces and their visitors [5]. Physical or virtual CHH objects can in- teract with people, environment, other objects, and transmitting the related information to users through multimedia facilities. Services become oriented to personalized recommendations when the system captures the event the visitor studies a CHH object. A set of “close” objects can be selected, ranked, and arranged to be provided to the visitor for subsequent study. Visitors are rich sources of new information about CHH objects, which is captured using annotations [1]. An added annotation enhances digital object memory, when the object stores data about itself and links other objects [17]. In a museum environment, object memory can store information about the prove- nance of the artifact, about its history, and the flow of comments generated by visitors while interacting with the artifact. 3 Smart Museum Information Services Smart museum extends the information archive function. The latter becomes supported with Internet-based services, acting as effective providers of CHH in- formation. Mobile multimedia, ambient intelligence, machine vision, augmented reality, and other IoT-related technologies lead to enhanced museum information services for effective CHH preservation, transmission, and research. 3.1 Possible ICT Use in Smart Museum The ICT use in a smart museum considers the following services that provide visitors or personnel with information for assisting their activity. Smart navigation: The museum visitors are notified (e.g., smarthones) about the recent situation in exhibition rooms (e.g., occupancy level, absent exhibits, or ongoing entertainments). The navigation become context-aware and subject to effective and personalized decision-making, e.g., see [14]. Quality evaluation: Visitors and their activity form a rich source for qualita- tive analytics on the museum expositions (e.g., most visited rooms, high-interest exhibits and information, or popular routes). The museum monitors its own function and evaluate its efficiency, so adapting and personalizing the service provision to the user’s needs, e.g., see [11]. Multimodal interface: Information provision is augmented using context and multimedia (e.g., visualization of interesting facts on a nearby screen, exhibit self- storytelling, or 3D modeling). That is, the transmission efficiency of preserved CHH knowledge to the visitor can be increased to the level similar to the personal assistance by an expert, e.g., see [18]. Collective intelligence: Involving visitors to CHH preservation within their museum activity (e.g., impression sharing, gamification, or collaborative esti- mates). The museum collection is enriched by information provided by visitors themselves, including knowledge created collaboratively. This property of social networks extends the museum to a cyber-physical-socio system, e.g., see [2]. 45 4 3.2 Basic Forms of Smart Museum Services Smart museum information services have the following forms in respect to the applicability needs in CHH domain. Preservation (and knowledge promotion): The museum digitally preserves information about the collected CHH entities, in addition to physical preserva- tion [15]. The linked-oriented forms of CHH information preservation showed their effectiveness in museums, e.g., see [14,12]. The collected information and its stored interlinked knowledge is subject to promotion when various audiences are advertized and involved to CHH knowledge consumption and use. Modern history is a special interest example. A museum visitor can be a participant of the event presented in the exhibition. People’s living memory is sometimes the only or at least the most accessible source of information about the events of unofficial history (family holidays and traditions, everyday household practices, ordinary citizen opinion to the events of “great history”, etc.). Transmission (and knowledge mastering): A museum environment provides a space for education activity [2]. In addition to self-education when a person (or group) individually makes CHH studies, enhanced education activity is possible, which includes thematic training or even pedagogical interaction. This activity needs context-aware information search for effective knowledge consumption and use. In particular, CHH information is visualized on nearby screens, and the users make interaction and storytelling with smart exhibits [18]. This form for CHH presentation makes the educational activity interesting for new categories of visitors, especially children and youth. Representatives of the elder generation also note that personal active participation in the selection of information for visualization makes the visit to the museum more interesting. The visitor concentrates more on the subjects of study and less on the supple- mentary information search function. Research (and knowledge enriching): A CHH researcher needs many infor- mation facts to create a fused picture from them and to interpret the value of this picture [9]. The initial information corpus is large, and the search cannot be done manually. In the transmission above, an information fragment is found. In the research, information fragments having potential knowledge are needed. The particular knowledge is created by an expert and enriches the museum collection. The search uses context relations between exhibits. Each individual exhibit has fragmentary information. The whole picture of a CHH phenomenon can be created when the relationship between individual facts is detected. Then new hypotheses can be built and grounded conclusions are drawn. 3.3 Semantic Integration The need of semantic integration of available CHH knowledge for creating smart museum information services has been already understood [16,12]. It supports creating new exhibitions, working with visitors on a personal or mini-group level, contributing to the realization of their expectations. 46 5 A mediation layer is introduced for semantic integration where knowledge is derived based on a distributed set of multiple data sources, including such services as DBpedia [4] and other services for semantic publishing, enrichment, search, and visualization [12,15]. We apply the semantic network model for inte- gration of digital CHH content [6,9,13]. The smart museum environment includes the following components. 1. Ontological model for structural representation of collected CHH exhibits as well as their various descriptions and relations with other objects. 2. The wiki system to transform the semantics from the collected records to the semantic network using experts and the ontological model. 3. Semantic data mining in the constructed semantic network to take into ac- count existing relations between collected exhibits and other CHH objects. An information service provides a search extend of the museum collection. Several most appropriate information facts are found for a given problem. It is close to the k-optimization approach (several top solutions are used). 4 Semantic Data Mining A semantic network is created on top of the museum collection—descriptions from the various information sources [9]. Formally, a semantic network is a di- rected graph G = (V, L) consisting of nodes (vertices set V ) representing domain objects and links (edges set L) representing semantic relations. The nodes corre- spond to physical exhibits (digitally virtualized) and digital exhibits (e.g., elec- tronic photos, scanned documents), associated events, persons, and other CHH objects. The links in L reflect interrelation of the objects. 4.1 Semantic Network The ontological model for the semantic network is defined by ontology O. First, O describes a system of concepts {Ci }ni=1 (ontology classes) such that any par- ticular node v ∈ V (ontology class object, instance or individual) belongs to one or more concepts. Second, O describes the interlinking structure for L, i.e., between which concepts a relation can be and possible types of such relations. The links represent the primary semantics. Third, O describes attributes that v ∈ V and l ∈ L may have to reflect additional semantics (e.g., keywords). Semantic network construction is implemented as a collective process [13]. On the one hand, many nodes v are straightforwardly derived from existing museum records (e.g., collected in the museum information system) or correspond to descriptions available in various open sources (e.g., web pages or photos in the Internet). On the other hand, for nodes v ∈ V the expert defines semantic relations (i.e., links l ∈ L) and their attributes. An information service needs to find k > 0 the most appropriate information facts. A fact can be a node v ∈ V , a link l ∈ L, or a connected graph structure s in G (e.g., a path from u to v can have CHH-valued interpretation for some 47 6 u, v ∈ V ). This data mining can be reduced to the ranking problem when rank values rv ≥ 0 or rl ≥ 0 are associated with nodes or links. The higher rank the better is appropriateness of the information. The rank of a connected graph structure is calculated based on ranks of the composed nodes and links. 4.2 Ranking Models We consider the following three classes of ranking methods [13]: 1) local ranking, 2) collaborative filtering, 3) structural ranking. Local ranking: Two or more objects are analyzed for similarity based on their content and overlapping of this content. In this case, the rank is computed in respect to some fixed node u ∈ V and reflects distance of other nodes from u: rv (u) = 1/ρ(u, v). For instance, if u and v have sets Ku and Kv of annotating keywords then the rank reflects the size of overlapping |Ku ∩ Kv |, i.e., the larger the number of shared keywords the higher is the similarity. In particular, if u is the recent exhibit that the visitor studies then the information service can provide the highest rank nodes v1 , . . . , vk as recommendation for the subsequent study. Collaborative filtering: This ranking model assumes that many users generate opinions about each CHH object. The opinions are transformed to some commu- nity based score rv∗ (normalized 0 ≤ rv∗ ≤ 1). Then the scores can be combined with other ranking requirements. For instance,   ∗ dv rv = αrv + (1 − α) 1 − , maxw∈W dw where W ⊂ V is nodes of potential interest for the visitor, dw > 0 is a physical reachability metric for node w, 0 ≤ α ≤ 1 is a tradeoff parameter between com- munity scores and reachability. In particular, if W is a set of points of interest for the visitor and dw is the time for the visitor to reach w from the current location then the information service can provide the highest rank nodes v1 , . . . , vk as recommendation for the next object to study. Structural ranking: This ranking model utilizes the connectivity properties of the semantic network G, similarly as it happens in the well-known PageRank algorithm for network analysis. For instance, node ranks ru for all u ∈ V can be (0) computed iteratively starting from some initial values ru : X ru(i+1) = α pvu rv(i) + (1 − α)πu , l=(u,v)∈L where pvu is weight of the link l = (u, v), 0 ≤ α ≤ 1 is the damping factor denoting the probability of following the connectivity structure of G, and π is a jump probability vector for all u ∈ V . In particular, if pvu are relative weight of CHH role of v to u then the information service can provide the highest rank nodes v1 , . . . , vk as recommendation for the most CHH-valued nodes to study. 48 7 5 Case Study The History Museum of Petrozavodsk State University (PetrSU) is a typical small museum oriented to everyday life history [9]. More than 10 digital displays of various sizes with changing images of photographs, documents, newspaper articles from different eras of more than 75-year PetrSU history. Transformable table makes it easy to change the Museum space, making it comfortable for different forms of collaborative work activities. Some displays show video and audio interviews with the teachers of the University in different years. Exhibits presented on windows show everyday life history of teachers, researchers, and students. Some exhibits, despite their advanced age, can be experienced directly in the room. Old movies about PetrSU life in the 1970s and 1980s provide the necessary cultural and historical atmosphere. Let us consider the three basic information services that we developed for use in this museum. Visual design and user interface details of the presented smart museum information services were demonstrated in [13]. The generic role of the smart museum services is summarized in Table 1. Visit service: The service constructs a personalized exposition of recom- mended exhibits for a visitor to study. Such a recommendation is a small set of selected objects from the presented ones in the museum exhibition room. This set VU is constructed from the available knowledge such that the set repre- sents the most interesting facts for the particular visitor u or their group U . This way, a visit program is constructed for a museum visitor before the visit. The service is also responsible for program adaptation during the visit depending on the preferences of the visitor and on the dynamically changing situation. The Museum is a point from which the guest often starts introduction with PetrSU. According to psychologists, the memorization of new information occurs when the knowledge is associated with the existing one. The service discovers facts in the biography of Museum visitors and intersects them with facts from the University history or with persons who worked (working) in PetrSU. The ed- ucational function becomes more effective, making an almost personal approach to each visitor and improving the whole PetrSU image. Table 1. The role of smart services in respect to museum collection Service Visit service Exhibition service Enrichment service Form Preservation Global scale structure Local scale structureCommunity-based collection Transmission Thematic navigation Personalized Involving the in CHH knowledge navigation in semantic community to the collection neighborhood for given activity exhibit Research Structured global Focused local view on Community resources view on CHH given fact within its utilization and knowledge collection neighborhood knowledge analysis 49 8 Visit service uses exhibit ranks based on local ranking (although other rank- ing models can be applied as well). In particular, rv = 1/ρ(U, v), where the distance ρ(U, v) shows semantic similarity of users U to the exhibit v ∈ V . The CHH interests of the user are included into the user profile and can be repre- sented in the node u ∈ U in G. Objects in VU can be ordered into the visit program (i.e., implementing smart navigation), which is in turn visualized on the public screen in the museum room or on personal mobile devices of the users (i.e., implementing multimodal interface). Exhibition service: The service shows selected descriptions and visual infor- mation about the studied exhibits on exhibition touch screens or on personal mobile devices of the visitors. In fact, the service creates a kind of virtualization when a physical exhibition is augmented with digital representation (i.e., imple- menting multimodal interface). As in Visit service, Exhibition service acts as a recommender since the screens show the recommended (most interesting) facts derived from the available CHH knowledge for the current context and situation. The service makes the Museum space more interesting and diverse. It attracts the most numerous group of museum visitors—young people. The widespread use of gadgets and effective extraction of information allow the student to use the services to find information important for her/his study. For recommendations local ranking can be used. In particular, rv = 1/ρ(u, v), where the distance ρ(u, v) shows the relation level of information fact v to the recently studied exhibit or fact u (for u, v ∈ V ). The recommendation assistance is online (i.e., implementing smart navigation). An effective visualization can employ the star graph model, where the internal node u has a small set of rays (leaves) to show the recommended facts and their interest level (rank). Enrichment service: The service supports modification (evolution) of the se- mantic network by museum personnel and visitors (i.e., implementing collective intelligence). A museum visitor can enrich descriptions of studied exhibits (e.g., adding annotations). A personal mobile device (e.g., smartphone) becomes a pri- mary access tool for this service. First, annotation is useful when the visitor adds descriptions about an object (e.g., facts from an eyewitness of the event), which is particularly important in everyday life history. Second, visitors can make the routine work on establishing known history-valued relations between objects. The visitor adds some relation (together with its description), and museum per- sonnel moderate the correctness and value. The museum collection is for modern history, preserving the events of the recent past and being constantly updated with modern exhibits. The collection often provokes visitors’ own memories of the University life. The service captures these memories and turns them into exhibits. Museum collection has many ex- hibits for which information needs clarification: the date of creation of photos are sometimes specified approximately, in group photos are only individual per- sons. There are many exhibits the purpose and work of which is not always fully understood. In particular, old devices and mechanisms that were used in the ed- ucational process and now lost their importance. The service supports collecting this clarification information from visitors and experts. 50 9 This service can use ranking for quality evaluation. In particular, let the initial semantic network G have ranks structure RG = {rG }. After community- based enrichment, the new semantic network is G∗ has another ranks structure RG∗ = {rG∗ }. Analysis of the difference between RG and RG∗ supports quanti- tative evaluation of the added value to the CHH knowledge the museum collects. 6 Conclusion This paper presented our view on smart museum information services in respect to their role for assisting cultural and historical heritage activity of both museum visitors and personnel. We reviewed the smart museum concept based on the recent progress in information and communication technology and in Internet of Things, in particular. We presented the basic forms of such services applicable for CHH activity within a smart museum. Our service development is based on applying the semantic network model for integrating heterogeneous CHH knowledge. Data mining in a semantic network is reduced to the ranking problem, which in turn is used for constructing the services. The effectiveness is discussed on the case study of the History Museum of PetrSU, where the proposed services are developed for the CHH preservation, transmission, and research needs. Acknowledgments. This research is financially supported by the Ministry of Education and Science of Russia within project # 2.5124.2017/8.9 of the basic part of state research assignment for 2017–2019. The work is implemented within the Government Program of Flagship University Development for Petrozavodsk State University in 2017–2021. References 1. Alzahrani, A.A., Loke, S.W., Lu, H.: An advanced location-aware physical annota- tion system: From models to implementation. Journal of Ambient Intelligence and Smart Environments 6(1), 71–91 (2014) 2. Amato, F., Moscato, V., Picariello, A., Colace, F., Santo, M.D., Schreiber, F.A., Tanca, L.: Big data meets digital cultural heritage: Design and implementation of scrabs, a smart context-aware browsing assistant for cultural environments. J. Comput. Cult. Herit. 10(1), 6:1–6:23 (Apr 2017) 3. Augusto, J., Callaghan, V., Cook, D., Kameas, A., Satoh, I.: Intelligent environ- ments: a manifesto. Human-centric Computing and Information Sciences 3(1), 1–18 (2013) 4. Bizer, C., Lehmann, J., Kobilarov, G., Auer, S., Becker, C., Cyganiak, R., Hell- mann, S.: DBpedia – a crystallization point for the Web of Data. Web Semantics: Science, Services and Agents on the World Wide Web 7, 154–165 (2009) 5. Chianese, A., Piccialli, F., Valente, I.: Smart environments and cultural heritage: A novel approach to create intelligent cultural spaces. Journal of Location Based Services 9(3), 209–234 (Jul 2015) 51 10 6. Kalus, M.: Semantic networks and historical knowledge management: Introduc- ing new methods of computer-based research. The Journal of the Association for History and Computing 10 (Dec 2007) 7. Korzun, D.: On the smart spaces approach to semantic-driven design of service- oriented information systems. In: Proc. 12th Int’l Baltic Conf. on Databases and Information Systems (DB&IS 2016). pp. 181–195. Springer International Publish- ing (Jul 2016) 8. Korzun, D., Marchenkov, S., Vdovenko, A., Petrina, O.: A semantic approach to designing information services for smart museums. International Journal of Em- bedded and Real-Time Communication Systems (IJERTCS) 7(2), 15–34 (2016) 9. Korzun, D., Varfolomeyev, A., Yalovitsyna, S., Volokhova, V.: Semantic infrastruc- ture of a smart museum: toward making cultural heritage knowledge usable and creatable by visitors and professionals. Personal and Ubiquitous Computing 21, 345–354 (Apr 2017) 10. Kuflik, T., Wecker, A., Lanir, J., Stock, O.: An integrative framework for extending the boundaries of the museum visit experience: linking the pre, during and post visit phases. Information Technology & Tourism 15(1), 17–47 (2015) 11. Lepouras, G., Naudet, Y., Antoniou, A., Lykourentzou, I., Tobias, E., Rompa, J.: Museum personalization based on gaming and cognitive styles: The BLUE exper- iment. Int. J. Virtual Communities Soc. Netw. 7(2), 1–30 (Apr 2015) 12. Mouromtsev, D., Haase, P., Pavlov, D., Cherny, E., Andreev, A., Spiridonova, A.: Towards the linked Russian heritage cloud: Data enrichment and publishing. In: The Semantic Web. Latest Advances and New Domains (ESWC2015). pp. 637–651. LNCS 9088, Springer (May 2015) 13. Petrina, O.B., Korzun, D.G., Volokhova, V.V., Yalovitsyna, S.E., Varfolomeyev, A.G.: Semantic approach to opening museum collections of everyday life history for services in Internet of Things environments. International Journal of Embedded and Real-Time Communication Systems (IJERTCS). Special Issue on Big Data Analytics and Intelligent Environments in Internet of Things 8(1), 31–44 (2017) 14. Ruotsalo, T., Haav, K., Stoyanov, A., Roche, S., Fani, E., Deliai, R., Makela, E., Kauppinen, T., Hyvonen, E.: SMARTMUSEUM: A mobile recommender system for the Web of Data. Web Semantics: Science, Services and Agents on the World Wide Web 20, 50–67 (May 2013) 15. Simou, N., Chortaras, A., Stamou, G., Kollias, S.: Enriching and Publishing Cul- tural Heritage as Linked Open Data, pp. 201–223. Springer International Publish- ing, Cham (2017) 16. Smirnov, A., Shilov, N., Kashevnik, A.: Ontology-based mobile smart museums service, approach for small & medium museums. In: Proc. of the 4th Intern. Conf. on Advances in Future Internet (AFIN 2012). pp. 48–54. IARIA (August 2012) 17. Torre, I.: Interaction with linked digital memories. In: Berkovsky, S., Herder, E., Lops, P., Santos, O.C. (eds.) Late-Breaking Results, Project Papers and Work- shop Proceedings of the 21st Conf. on User Modeling, Adaptation, and Personal- ization (UMAP 2013). Proc. Workshop on Personal Access to Cultural Heritage (PATCH 2013). CEUR Workshop Proceedings, vol. 997, pp. 80–87. CEUR-WS.org (2013) 18. Vassilakis, C., Poulopoulos, V., Antoniou, A., Wallace, M., Lepouras, G., Nores, M.L.: exhiSTORY: Smart exhibits that tell their own stories. Future Generation Computer Systems 81, 542–556 (2018) 52