In this short paper we report on work in progress in the research project \Chatbot in the museum" (ChiM). ChiM develops a practicable technical solution for the use of chatbots in the museum environment. The paper outlines conceptual work conducted so far, including the comprehension of three important research topics explored in ChiM, namely information processing, multimodal intent detection and dialog management for museum chatbots.
A chatbot is a computer program that attempts to simulate the conversation of a
human being via text or voice interactions [
Conversations with museum visitors showed that they often have speci c questions about certain objects. Classical audioguides can not answer speci c questions. In the museum a chatbot could be the expert you can take with you, answer questions and provide further information. Fig. 1 exempli es how the conversational interaction between a visitor and the chatbot could look like.
The aim of ChiM is to explore the usability of a chatbot as an interactive system for knowledge and learning, as well as for e ective access and the comprehensible presentation of museum information. A central question is therefore how the existing information must be structured in order to relate to the visitors' questions. ChiM develops new solutions for providing information and explores how the latest research results in the eld of intention detection and dialog management can be utilized for chatbots in the museum context. Speci cally, the following research elds of human-technology interaction are to be investigated: { Information processing : adaptation of the existing content creation process for classical audio and media guides, to be more \knowledge-based" for the chatbot. { Multimodal intention detection: linguistic or text-based input processing combined with image and other exhibition sensor (e.g. beacons) processing. { Dialog management for guided tours : intelligent dialog strategies to determine context-relevant information and personalized information.
In this paper conceptual work in the ChiM research elds information processing, multimodal intent detection and dialog management for museum chatbots is outlined. In the next section related work in chatbot relevant elds is sketched. Section 3 describes the so far developed concepts for ChiM. Section 4 describes ongoing work on use case development. Conclusion and future work are described in Section 5. 2
Current research in the eld of interactive museum guides covers a wide range
of approaches, from beacons controlling the presentation [
for ordering a pizza, weather report, ight booking, shopping, etc.) [
ChiM investigates hybrid methods for intention detection and dialog management for the museum sector. A newly developed chatbot-based museum guide usually represents a new knowledge domain for which training data is not yet available. During the process of creating audio and multimedia guides, many data are generated, which can also be used as training data if appropriate transcription and indexing methods are applied. ChiM's approach is to combine the development of a hybrid approach for intention detection and dialog management with the creation process of museum tours, using the data generated by authors and editors as training material for a statistical model. Domain-speci c knowledge components, which can not or can hardly be statistically mapped, will be realized through dialog grammars. In addition, the consideration of further information channels (e.g. image recognition, exhibition sensor technology) enables multimodal intent detection. Subsections 3.1-3.3 summarize the so far developed concepts for the research areas explored in ChiM. Subsection 3.4 explains the iterative realization of ChiM. 3.1
In contrast to classical audio and multimedia guides, information for a chatbot system must be structured di erently. The entire existing text creation process must be converted into a more \knowledge-based" appraoch. ChiM needs (semi) structured data to allow for a exible interaction with the content. For data preparation, approaches such as taxonomies and the use of digital asset management architectures such as Fedora Commons4 will be integrated into editors' work ows. Further, it will be investigated whether and how the existing data for museum guides and their contents can be prepared in such a way that they can serve as training material. 3.2
Detecting user intent in the museum environment is a complex, multimodal
process. Visitors can interact with text or voice input. Thus, on the one hand,
the linguistic or text-based input of the user must be processed. On the basis
of historical data and the results of the information processing, statistical and
rule-based procedures will be evaluated and used. On the other hand, the
consideration of further information channels, such as image processing and existing
exhibition sensors (e,g. beacons for localization), is of particular interest in a
museum context. For this, the MMIR (Multimodal Mobile Interaction and
Rendering) framework will be used and extend [
The individual steps from intention detection over the retrieval of the information to the provision of information are continuously carried out in a dialog between the user and the chatbot. As the core of the chatbot, an intelligent dialog management will process the multimodal intention and determine the system reaction to o er personalized information. The dialog should always be e ective, intuitive and continued with the best possible user experience. The input/output modalities will be adapted to the situation using text, speech or multimedia elements.
More speci cally, the information presented to the users is decided from
the intention, the dialog history, the knowledge model from the information
processing and the general context. The processing of the dialog management will
be hybrid: from historical data typical museum guide sequences are learned. At
the same time, dialog rules will be created and the two approaches combined [
The realization takes place in two iterations using a user-centered and participative design approach. In the rst iteration, the editors will prepare the knowledge for an exhibition. At the same time the basic ChiM functionality will be investigated with focus groups in an iterative UX process and tested in the lab with potential users. In the second iteration, the ChiM process will be implemented within further exhibition guides and evaluated with users in the context of eld studies. 4
As part of our ongoing work we elaborate on the proposed input and output modalities of the system and exemplify the goal of our approach with respect to the user by outlining rst use case ideas which are related to multimodal interaction. As a precondition for all use cases it can be assumed that the chatbot is installed on a smartphone which has the necessary technical speci cations, i.e. microphone, camera, and iBeacon, or respectively bluetooth low energy.
As main input modalities touch input on a virtual keyboard and alternatively speech input enabled by automatic speech recognition (ASR) will be utilized to interact with the chatbot. A smartphone camera will be used for computer vision in order to recognize exhibits or other objects relevant for a speci c exhibition. Further the proximity to exhibits will be detected by beacons and processed analog to the context.
The output modalities will include visual, auditory and tactile feedback. The visual information comprises mainly the dialog with the chatbot and media content like images and videos. The chatbot will make use of speech synthesis to create auditory feedback for the system promts within the user-chatbot dialog. System promts will be generated for meta communication, i.e. system con guration, as well as for speci c information coming from the knowledge model speci c to the exhibition. Further auditory content consists of recorded audio material (also in combination with image or video) as in classical multimedia guides. Tactile feedback can be used for alerts, e.g. if the user gets closer to the next exhibit of a tour.
With regard to the context in which this interaction will take place a number of factors have to be considered. At the place itself, i.e. a museum or an exhibition, the acoustic characteristics can be very di erent, as the exhibition areas can be located in both large halls and smaller rooms. Further, people in the surrounding area can on the one hand produce noise which is harmful for ASR, on the other hand talking to the chatbot could distract other visitors. Within an exhibition the auditory signal should therefore be received by the user via headphones. It can further be necessary to avoid the usage of speech input if other visitors are located right next to someone. The system should therefore always provide touch input on a virtual keyboard as an alternative input mode. To determine relevant information not only the actual user input but also the last user inputs, questions, or interactions should be considered to enable a personalized request.
Based on these assumptions we so far generated the following ideas for multimodal use cases: { Sequential usage of touch or speech with independent fusion : either touch screen or speech input can be used to enter text based questions about the exhibits to the chatbot. { Sequential usage of computer vision and textual information with combined fusion: the camera can be used to enter visual information that is recognized by means of computer vision. If an image was recognized speci c information about the the corresponding exhibit can be asked, e.g. after taking a picture from a sculpture the user can use touch screen or speech input to ask "Who made this work?". { Sequential usage of textual information and exhibition sensors with combined fusion: after speci c textual input via touch screen or speech input, e.g. "Where does it go on?", exhibition sensor processing (e.g. beacons) can be used to guide the user to the closet exhibit on the tour. 5
The development of a practical technical solution for the use of chatbots in the museum environment represents a challenging task: the editing process for audio and media guides is a highly specialized process, and the extension to knowledgebased approaches for the realization of a chatbot for museums has to be carried out. The interplay of existing approaches for intention detection and dialog management opens up a number of research questions, including how chatbots can be used in complex environments such as museums. A solution for hybrid knowledge processing and the technical implementation of information processing, multimodal intent detection and dialog management for museum chatbots will be explored in ChiM. Di erent multimodal use cases will be studied. The success of the project will be assessed by a demonstrator implemented iteratively in two development phases. To ensure the acceptance of ChiM the overall system will be designed, developed, analyzed and optimized in cooperation with museums and visitors. Focus groups and eld studies in various museums will be conducted. 6
We thank the Stadel Musem Frankfurt, the Germanische Nationalmuseum Nurnberg and the LVR- LandesMuseum Bonn for their interest in supervising, testing and evaluating the ChiM developments.