An intelligent mobile museum visitors' guide is a canonical case of a context-aware mobile system. Museum visitors move in the museum, looking for interesting exhibits, and wish to acquire information to deepen their knowledge and satisfy their interests. A smart context-aware mobile guide may provide the visitor with personalized relevant information from the vast amount of content available at the museum, adapted for his or her personal needs. Earlier studies relied on using sensors for location-awareness and interest detection. This work explores the potential of mobile eye-tracking and vision technology in enhancing the museum visit experience. Our hypothesis is that the use of the eye tracking technology in museums' mobile guides can enhance the visit experience by enabling more intuitive interaction. We report here on satisfactory preliminary results from examining the performance of a mobile eye tracker in a realistic setting - the technology has reached a reliable degree of maturity that can be used for developing a system based on it.
Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes.
Tsvi Kuflik
The University of Haifa Mount Carmel, Haifa, 31905
+97248288511 tsvikak@is.haifa.ac.il Falk and Dierking [2000] and Falk [2009] tried to answer the question of what do visitors remember from their visit and what factors seemed to most contribute to visitors' forming of long-terms memories: “when people are asked to recall their museum experiences, whether a day or two later or after twenty or thirty years, the most frequently recalled and persistent aspects relate to the physical context-memories of what they saw, what they did, and how they felt about these experiences.”. Stock et al. [2009], and Dim and Kuflik [2014] explored the potential of novel, mobile technology in identifying visitors behavior types in order to consider what/how/when to provide them with relevant services. A key challenge in using mobile technology for supporting museum visitors' is figuring out what they are interested in. This may be achieved by tracking where the visitors are and the time they spend there [Yalowitz and Bronnenkant, 2009]. A more challenging aspect is finding out what exactly they are looking at [Falk and Dierking, 2000]. Given todays' mobile devices, we should be able to gain access seamlessly to information of interest, without the need to take pictures or submit queries and look for results, which are the prevailing interaction methods with our mobile devices. As we move towards "Cognition-aware computing" [Bulling and Zander 2014], it becomes clearer that eye-gaze based interaction may play a major role in human-computer interaction before/until brain computer interaction methods will become a reality [Bulling et al. 2012]. The study of eye movements started long almost 100 years ago, Jacob and Karn [2003] presented a brief history of techniques that were used to detect eye movements, the major works dealt with usability researches, one of the important works started in 1947 by Fitts and his colleagues [Fitts et al. 1950] when they began using motion picture cameras to study the movements of pilots’ eyes as they used cockpit control and instruments to land an airplane. “It is clear that the concept of using eye tracking to shed light on usability issues has been around since before computer interfaces, as we know them” [Jacob and Karn 2003]. Certain mobile eye tracking devices that enables to detect what someone is looking at and stores the data for later use and analysis, have been developed and could be found in the market nowadays [Hendrickson et al. 2014]. In recent years, eye tracking and image based object recognition technology have reached a reliable degree of maturity that can be used for developing a system based on it, precisely identifying what the user is looking at [Kassner et al. 2014]. We shall refer to this field by reviewing techniques for image matching and extend them for location-awareness use and we will follow the approach of “What you look at is what you get” [Jacob 1991].
With the advent of mobile and ubiquitous computing, it is time to explore the potential of this technology for natural, intelligent interaction of users with their smart environment, not only in specific tasks and uses, but for a more ambitious goal of integrating eye tracking into the process of inferring mobile users’ interests and preferences for providing them with relevant services and enhancing their user models, an area that received little attention so far. This work aims at exploring the potential of mobile eye tracking technology in enhancing the museum visit experience by integrating and extending these technologies into a mobile museum visitors' guide system, so to enable using machine vision for identifying visitors' position and their object of interest in this place, as a trigger for personalized information delivery. 2. BACKGROUND 2.1 Museum visitors and their visit experience Understanding who visits the museum, their behaviors and the goal of the visit can play an important role in the design of museums’ mobile guide (and other technologies) that enhances the visit experience, “the visitors’ social context has an impact on their museum visit experience. Knowing the social context may allow a system to provide socially aware services to the visitors.” [Bitgood 2002; Falk 2009; Falk and Dierking 2000; Leinhardt and Knutson 2004; Packer and Ballantyne 2005]. Falk [2009] argued that many studies have been done on who visits museums, what visitors do in the museum and what visitors learn from the museum, and tried to understand the whole visitor and the whole visit experience as well as after the visit. Furthermore, he proposed the idea of visitors "identity" and identified five, distinct, identity-related categories: • Explorers: Visitors who are curiosity-driven with a generic interest in the content of the museum. They expect to find something that will grab their attention and fuel their learning. • Facilitators: Visitors who are socially motivated. Their visit is focused on primarily enabling the experience and learning of others in their accompanying social group. • Professional/Hobbyists: Visitors who feel a close tie between the museum content and their professional or hobbyist passions. Their visits are typically motivated by a desire to satisfy a specific content-related objective. • Experience Seekers: Visitors who are motivated to visit because they perceive the museum as an important destination. Their satisfaction primarily derives from the mere fact of having ‘been there and done that’. • Rechargers: Visitors who are primarily seeking to have a contemplative, spiritual and/or restorative experience. They see the museum as a refuge from the work-a-day world or as a confirmation of their religious beliefs.
In addition, he argued that the actual museum visit experience is strongly shaped by the needs of the visitor’s identity-related visit motivations, and the individual’s entering motivations creates a basic trajectory for the visit, though the specifics if what the visitor actually sees and does are strongly influenced by the factors described by the Contextual Model of Learning: • Personal Context: The visitor’s prior knowledge, experience, and interest. • Physical Context: The specifics of the exhibitions, programs, objects, and labels they encounter. • Socio-cultural Context: The within-and between-group interactions that occur while in the museum and the visitor’s cultural experiences and values.
Nevertheless the visitor perceives his or her visit experience to be satisfying if this marriage of perceived identity-related needs and museum affordance proves to be well-matched. Hence, considering the use of technology for supporting visitors and enhancing the museum visit experience, it seems that these aspects need to be addressed by identifying visitors' identity and providing them relevant support. 2.2 Object recognition and image matching Modern eye trackers usually record video by a front camera of the scenes for further analysis [Kassner et al. 2014]. Object recognition is a task within computer vision of finding and identifying objects in an image or video sequence. Humans recognize a multitude of objects in images with little effort, despite the fact that the image of the objects may vary somewhat in different viewpoints, in many different sizes and scales or even when they are translated or rotated. Objects can even be recognized when they are partially obstructed from view. This task is still a challenge for computer vision systems [Pinto et al. 2008]. Many approaches to the task have been implemented over multiple decades. For example, diffusing models to perform image-to-image matching [Thirion 1998], parametric correspondence technique [Barrow 1977] and The Adaptive Least Squares Correlation [Gruen 1985] were presented as a techniques for image matching. Techniques from [Naphade et al. 1999], [Hampapur et al. 2001] and [Kim et al. 2005] were presented for image sequence matching (video stream). A related field is visual saliency or saliency detection, “it is the distinct subjective perceptual quality which makes some items in the world stand out from their neighbors and immediately grab our attention.” [Laurent 2007]. Goferman et al. [2012] proposed a new type of saliency which aims at detecting the image regions that represent the scene. In our case, we can exploit the use of eye tracking to detect salience in an efficient way since we have fixation points representing points of interests in a scene. 3. RELATED WORK As mentioned above, many studies were conducted in detecting eye movements before considering their integration with computer interfaces, as we know them today. The studies have been around HCI and usability and techniques were presented that can be extended for further eye tracking studies and not just in the field of HCI. Jacob [1991] presented techniques for local calibrating of an eye tracker, which is a procedure of producing a mapping of the eye movements’ measures and wandering in the scene measures. In addition, he presented techniques for fixation recognition with respect to extracting data from noisy, jittery, error-filled stream and for addressing the problem of "Midas touch” where people look at an item without having the look “mean” something. Jacob and Karn [2003] presented a list of promising eye tracking metrics for data analysis: • Gaze duration - cumulative duration and average spatial location of a series of consecutive fixations within an area of interest. • Gaze rate – number of gazes per minute on each area of interest. • Number of fixation on each area of interest. • Number of fixation, overall. • Scan path – sequence of fixations. • Number of involuntary and number of voluntary fixations (short fixations and long fixations should be defined well in term of millisecond units).
Using handheld devices as a multimedia guidebook in museums has led to improvement in the museum visit experience. Researches have confirmed the hypothesis that a portable computer with an interactive multimedia application has the potential to enhance interpretation and to become a new tool for interpreting museum collections [Evans et al. 2005, Evans et al. 1999, Hsi 2003]. Studies about integration of multimedia guidebooks with eye tracking have already been made in the context of museums and cultural heritage sites. Museum Guide 2.0 [Toyama et al. 2012] was presented as a framework for delivering multimedia content for museum’s visitors which runs on handheld device and uses the SMI viewX eye tracker and object recognition techniques. The visitor can hear audio information when detecting an exhibit. A users' study was conducted in a laboratory setting, but not in a real museum. We plan to extend this work by integrating an eye tracker into real museum visitors' guide system and experiment it is realistic setting.
Brône et al. [2011] have implemented effective new methods for analyzing gaze data collected with eye-tracking device and how to integrate it with object recognition algorithms. They presented a series of arguments why an object-based approach may provide a significant surplus, in terms of analytical precision. Specifically they discussed solutions in order to reduce the substantial cost of manual video annotation of gaze behavior, and have developed a series of proof-of-concept case studies in different real world situations, each with its own challenges and requirements. We plan to use their lessons in our study. Pfeiffer et al. [2014] presented "EyeSee3D", where they combined geometric modelling with inexpensive 3D marker tracking to align virtual proxies with the real-world objects. This allowed classifying fixations on objects of interest automatically while supporting a free movement of the participant. During the analysis of the accuracy of the pose estimation they found that the marker detection may fail from several reasons: First, sometimes the participant looked sideways and there simply was no marker within view. More often, however, swift head movements or extreme position changes were causing these issues. Ohm et al. [2014] tried to find where people look at, when navigating in a large scale indoor environment, and what objects can assist them to find their ways. They conducted a user study and assessed the visual attractions of objects with an eye tracker. Their findings show that functional landmarks like doors and stairs are most likely to be looked at. In our case we can use these landmarks as reliable points of interest that can be used for finding the location of the visitor in the museum. Beugher et al. [2014] presented a novel method for the automatic analysis of mobile eye-tracking data in natural environment for object recognition. The obtained results were satisfactory for most of the objects. However, a large scale variance results in a lower detection rate (for objects which were looked at both from very far away and from close by.) Schrammel et al. [2011, 2014] studied attentional behavior of users on the move. They discussed the unique potential and challenges of using eye tracking in mobile settings and demonstrated the ability to use it to study the attention on advertising media in two different situations: within a digital display in public transport and towards logos in a pedestrian shopping street as well as ideas about a general attention model based on eye gaze. Kiefer et al. [2014] also explored the possibility of identifying users’ attention by eye tracking in the setting of tourism – when a tourist gets bored looking at a city panorama – this scenario may be of specific interest for us as locations or objects that attracted more or less interest may be used to model user's interest and trigger further services/information later on. Nakano and Ishii (2010) studied the use of eye gaze as an indicator for user engagement, trying also to adapt it to individual users. Engagement may be used as an indicator for interest and the ability to adapt engagement detection to individual users may enable us also to infer interest and build/adapt a user model using this information. Furthermore, Ma et al. [2015] demonstrated an initial ability to extract user models based on eye gaze of users viewing videos. Xu et al. [2008] also used eye gaze to infer user preferences in the content of documents and videos by the users attention as inferred from gaze analysis (number of fixations on word/image).
As we have seen, there is a large body of work about
monitoring and analyzing users' eye gaze in general and also
in cultural heritage setting. Moreover, the appearance of
mobile eye trackers opens up new opportunities for research in
mobile scenarios. It was also demonstrated in several
occasions that eye gaze may be useful in enhancing a user
model, as it may enable to identify users' attention (and
interests). Considering mobile scenarios, when users also carry
smartphones - equipped with various sensors - implicit user
modeling can take place by integrating signals from various
sensors, including the new sensor of eye-gaze for better
modeling the user and offering better personalized services. So
far sensors like GPS, compass, accelerometers and voice
detectors were used in modeling users' context and interests,
During the study we ran into several practical problems. The Pupil Dev eye tracker that we are using is not fitted for every person. The device consists of two cameras, the first for delivering the scene and the second directed to the right eye for detecting fixations. In some cases when the device is not fitted correctly, the vision range got smaller and parts of the pupil got out from the capture frame (see figure 3 for example). As a consequence no fixations were detected. Another limitation was that when using the eye tracker with tall participants, they have to step back from the object which negatively affects the accuracy. 6.3 User study 2: Look at an exhibit In this study we examined the accuracy of the eye tracker in a realistic setting. One participant (1.79m tall) was asked to look at exhibits at the Hecht museum. Several exhibits where chosen with different factors and constraints (see figure 4, 5, and 6). The main constraint in this case is the distance from the exhibit since the visual range gets larger when the distance grows, and mainly we have to cover all the objects that we are interested in. Table 2 presents the objects height from the floor and the distance of the participant from the object. The next step was to examine fixations accuracy after making sure that the participant is standing in a proper distance. The participant was asked to look at different points in the exhibit/scene. In the gallery exhibits, the scan path has been set to be the four corners of the picture and finally the center of it. Regarding the vitrine exhibits, for each jug one point at the center has been defined It’s important to note that the heights/distances relation is for visual range (having the objects in the frame of the camera) and not for fixations detections. Since missed fixations could be as a result of a set of constraints and not the distance from the object, thing that we have not examined yet. 7. SYSTEM DESIGN A smart context-aware mobile museum visitors' guide may provide the visitor with personalized relevant information from the vast amount of content available at the museum, adapted for his or her personal needs. Furthermore, the system may provide recommendations and location-relevant information. However, the potential benefit may also have a cost, the notifications may interrupt the user in the current task and be annoying in the wrong context. Beja et al. [2015] examined the effect of notifications in a special leisure scenario - a museum visit. Following Beja et al [2015], we will consider three different scenarios: I. The Visitor is looking at an exhibit. The region of interest will be defined as the region from the scene around the gaze fixation point. Then object matching procedure will be applied (see section 8). It will enable us to determine both the visitor’s position and the object of interest. II.
III.
The visitor is looking at the tablet. This could be done in two ways: 1) the visitor is watching multimedia information, in this scenario there is nothing to do for him. 2) The visitor may need service from the system or a recommendation, so it is the right time to deliver him. The visitor is wandering in the museum. According to Beja et al. [2015], it is the best time for sending notifications. As a basic system we will use the PIL museum visitor's guide system [Kuflik et al 2012; Lanir et al. 2013]. The system is a context aware, mobile museum visitors' guide system. Its positioning mechanism is based on proximity based RF technology that enables to identify the visitor's position – when the visitor is near a point of interest. As vision is the main sense for gathering information, we plan to replace the system's positioning component with an eye-tracker based positioning and object of interest identification component. Hence we will enhance the positioning system by providing the system the ability to pin-point the object of interest. The rest of the system will remain unchanged. Having these two versions of systems will enable us to compare and evaluate the benefits of the eye-tracker as a positioning and pointing device in the museum. 8. OBJECT MATCHING PROCEDURE 8.1 Data-set preparation A set of images of the exhibits will be taken, each image may contain one or more objects. Each image will be given a distinct label value and size of region around the object (in terms of width and height – rectangular shape). 8.2 Object matching The matching procedure will be done in three steps: 1. Eye-tracker scene camera frame is taken (figure 7) and image-to-image matching applied. The result is an image with labeled regions in the current scene’s frame (figure 8). 2. Mapping transformation – We need to transform the fixation point in the eye-tracker scene camera to a suitable/matched point in the image that we got in step one (image from the data-set with labeled regions), since the viewpoint of the objects can be different from this in the data set. For example one image is rotated relative to the other or one is zoomed in/out as a result of standing in different distance from the object when the data-set image was taken. 3. Finding the object - This is step is simple since we have a mapped fixation points and labeled regions. What remains is determining for which object the point does it relates (or it relates to nothing). 9. DISCUSSION We conducted these small-scale user studies in order to gain initial first-hand experience with the eye-tracker in a realistic setting. Furthermore, we tried to clarify which exhibits are appropriate to be included in our future study and, given the limitation of the device, what portion of the museum exhibits may be included in general. Not surprisingly, we got 100% accuracy rate when we examined the device in the art wing since all the pictures are placed in ideal height. Regarding the archeological wing, it is considerably more challenging environment, since objects are placed in different heights and have unequal sizes. As a result the visitor may have to stand far away from the objects in order to get them into the eyetracker front camera frame, a fact that can negatively affect the visit experience. In the case of archeological wing we approximate that about 60% of the exhibits may be detectable with the current device. Regarding the low-height exhibits we don’t know yet whether they can be considered or not. More challenging exhibits are these that are placed in harsh light conditions or placed in low height (see figure 9 for example) and/or these that are too large to fit in one frame (see figure 10 for example). 10. CONCLUSIONS AND FUTURE WORK This paper presents a work-in-progress that aims at exploring the potential contribution of the mobile eye tracking technology in enhancing the museum visit experience. For that we have done small-scale experiments in order to get an understanding of the performance of the system in realistic setting. We got satisfactory results from these studies and an understanding of the limitations of the equipment. The next step in the study is to design and build a museum mobile guide that extends the use of mobile eye tracking as a tool for identifying the visitor position and points of interests. We will use the eye-tracker scene camera captures and the collected gaze data to develop a technique for location-awareness. The system will run on tablet, and the multimedia content will be delivered to the participants by listening to audio guide via earphones or by watching slides. Furthermore, knowing exactly where the visitor look in the scene (specific object) will allow us to deliver personalized information. Our research will be a supplement to the nowadays mobile museum guide that uses location-awareness technology and techniques that enhances the visit experience. The system can also be extended and used in other venues such as outdoors cultural heritage sites as well as shopping centers/markets after further validation.