Exploring the potential contribution of mobile eye-tracking technology in enhancing the museum visit experience Moayad Mokatren Tsvi Kuflik The University of Haifa The University of Haifa Mount Carmel, Haifa, 31905 Mount Carmel, Haifa, 31905 +97248288511 +97248288511 moayad.mokatren@gmail.com tsvikak@is.haifa.ac.il ABSTRACT Falk and Dierking [2000] and Falk [2009] tried to answer the An intelligent mobile museum visitors’ guide is a canonical question of what do visitors remember from their visit and case of a context-aware mobile system. Museum visitors move what factors seemed to most contribute to visitors' forming of in the museum, looking for interesting exhibits, and wish to long-terms memories: “when people are asked to recall their acquire information to deepen their knowledge and satisfy museum experiences, whether a day or two later or after their interests. A smart context-aware mobile guide may twenty or thirty years, the most frequently recalled and provide the visitor with personalized relevant information persistent aspects relate to the physical context-memories of from the vast amount of content available at the museum, what they saw, what they did, and how they felt about these adapted for his or her personal needs. Earlier studies relied on experiences.”. Stock et al. [2009], and Dim and Kuflik [2014] using sensors for location-awareness and interest detection. explored the potential of novel, mobile technology in This work explores the potential of mobile eye-tracking and identifying visitors behavior types in order to consider vision technology in enhancing the museum visit experience. what/how/when to provide them with relevant services. Our hypothesis is that the use of the eye tracking technology in museums’ mobile guides can enhance the visit experience A key challenge in using mobile technology for supporting by enabling more intuitive interaction. We report here on museum visitors' is figuring out what they are interested in. satisfactory preliminary results from examining the This may be achieved by tracking where the visitors are and performance of a mobile eye tracker in a realistic setting – the the time they spend there [Yalowitz and Bronnenkant, 2009]. technology has reached a reliable degree of maturity that can A more challenging aspect is finding out what exactly they are be used for developing a system based on it. looking at [Falk and Dierking, 2000]. Given todays' mobile devices, we should be able to gain access seamlessly to Author Keywords information of interest, without the need to take pictures or Mobile guide; Mobile eye tracking; Personalized information; submit queries and look for results, which are the prevailing Smart environment; Context aware service. interaction methods with our mobile devices. As we move towards "Cognition-aware computing" [Bulling and Zander ACM Classification Keywords 2014], it becomes clearer that eye-gaze based interaction may H.5.2. Input devices and strategies (e.g., mouse, touchscreen) play a major role in human-computer interaction before/until brain computer interaction methods will become a reality 1. INTRODUCTION [Bulling et al. 2012]. The study of eye movements started long The museum visit experience has changed over the last two almost 100 years ago, Jacob and Karn [2003] presented a brief decades. With the progress of technology and the spread of history of techniques that were used to detect eye movements, handheld devices, many systems were developed to support the major works dealt with usability researches, one of the the museum visitor and enhance the museum visit experience. important works started in 1947 by Fitts and his colleagues The purpose of such systems was to encourage the visitors to [Fitts et al. 1950] when they began using motion picture use devices that provide multimedia content rather than use cameras to study the movements of pilots’ eyes as they used guide books, and as a consequence focus of the exhibits cockpit control and instruments to land an airplane. “It is clear instead of flipping through pages in a guide book, as surveyed that the concept of using eye tracking to shed light on usability in [Ardissono et al. 2012]. issues has been around since before computer interfaces, as we Understanding the museum visitors’ motivations plays a know them” [Jacob and Karn 2003]. Certain mobile eye crucial role in the development and designing of systems that tracking devices that enables to detect what someone is support their needs and could enhance their visit experience. looking at and stores the data for later use and analysis, have been developed and could be found in the market nowadays [Hendrickson et al. 2014]. In recent years, eye tracking and image based object recognition technology have reached a reliable degree of maturity that can be used for developing a Copyright © 2016 for this paper by its authors. Copying permitted for system based on it, precisely identifying what the user is private and academic purposes. looking at [Kassner et al. 2014]. We shall refer to this field by reviewing techniques for image matching and extend them for location-awareness use and we will follow the approach of creates a basic trajectory for the visit, though the specifics if “What you look at is what you get” [Jacob 1991]. what the visitor actually sees and does are strongly influenced by the factors described by the Contextual Model of Learning: With the advent of mobile and ubiquitous computing, it is time to explore the potential of this technology for natural, • Personal Context: The visitor’s prior knowledge, intelligent interaction of users with their smart environment, experience, and interest. not only in specific tasks and uses, but for a more ambitious • Physical Context: The specifics of the exhibitions, goal of integrating eye tracking into the process of inferring programs, objects, and labels they encounter. mobile users’ interests and preferences for providing them • Socio-cultural Context: The within-and between-group with relevant services and enhancing their user models, an interactions that occur while in the museum and the visitor’s area that received little attention so far. This work aims at cultural experiences and values. exploring the potential of mobile eye tracking technology in enhancing the museum visit experience by integrating and Nevertheless the visitor perceives his or her visit experience to extending these technologies into a mobile museum visitors' be satisfying if this marriage of perceived identity-related guide system, so to enable using machine vision for needs and museum affordance proves to be well-matched. identifying visitors' position and their object of interest in this Hence, considering the use of technology for supporting place, as a trigger for personalized information delivery. visitors and enhancing the museum visit experience, it seems that these aspects need to be addressed by identifying visitors' 2. BACKGROUND identity and providing them relevant support. 2.1 Museum visitors and their visit experience Understanding who visits the museum, their behaviors and the 2.2 Object recognition and image matching goal of the visit can play an important role in the design of Modern eye trackers usually record video by a front camera of museums’ mobile guide (and other technologies) that the scenes for further analysis [Kassner et al. 2014]. Object enhances the visit experience, “the visitors’ social context has recognition is a task within computer vision of finding and an impact on their museum visit experience. Knowing the identifying objects in an image or video sequence. Humans social context may allow a system to provide socially aware recognize a multitude of objects in images with little effort, services to the visitors.” [Bitgood 2002; Falk 2009; Falk and despite the fact that the image of the objects may vary Dierking 2000; Leinhardt and Knutson 2004; Packer and somewhat in different viewpoints, in many different sizes and Ballantyne 2005]. Falk [2009] argued that many studies have scales or even when they are translated or rotated. Objects can been done on who visits museums, what visitors do in the even be recognized when they are partially obstructed from museum and what visitors learn from the museum, and tried to view. This task is still a challenge for computer vision systems understand the whole visitor and the whole visit experience as [Pinto et al. 2008]. Many approaches to the task have been well as after the visit. Furthermore, he proposed the idea of implemented over multiple decades. For example, diffusing visitors "identity" and identified five, distinct, identity-related models to perform image-to-image matching [Thirion 1998], categories: parametric correspondence technique [Barrow 1977] and The Adaptive Least Squares Correlation [Gruen 1985] were • Explorers: Visitors who are curiosity-driven with a generic presented as a techniques for image matching. Techniques interest in the content of the museum. They expect to find from [Naphade et al. 1999], [Hampapur et al. 2001] and [Kim something that will grab their attention and fuel their et al. 2005] were presented for image sequence matching learning. (video stream). A related field is visual saliency or saliency • Facilitators: Visitors who are socially motivated. Their visit detection, “it is the distinct subjective perceptual quality which is focused on primarily enabling the experience and makes some items in the world stand out from their neighbors learning of others in their accompanying social group. and immediately grab our attention.” [Laurent 2007]. • Professional/Hobbyists: Visitors who feel a close tie Goferman et al. [2012] proposed a new type of saliency which between the museum content and their professional or aims at detecting the image regions that represent the scene. In hobbyist passions. Their visits are typically motivated by a our case, we can exploit the use of eye tracking to detect desire to satisfy a specific content-related objective. salience in an efficient way since we have fixation points • Experience Seekers: Visitors who are motivated to visit representing points of interests in a scene. because they perceive the museum as an important destination. Their satisfaction primarily derives from the 3. RELATED WORK mere fact of having ‘been there and done that’. As mentioned above, many studies were conducted in • Rechargers: Visitors who are primarily seeking to have a detecting eye movements before considering their integration contemplative, spiritual and/or restorative experience. They with computer interfaces, as we know them today. The studies see the museum as a refuge from the work-a-day world or as have been around HCI and usability and techniques were a confirmation of their religious beliefs. presented that can be extended for further eye tracking studies In addition, he argued that the actual museum visit experience and not just in the field of HCI. Jacob [1991] presented is strongly shaped by the needs of the visitor’s identity-related techniques for local calibrating of an eye tracker, which is a visit motivations, and the individual’s entering motivations procedure of producing a mapping of the eye movements’ measures and wandering in the scene measures. In addition, he sideways and there simply was no marker within view. More presented techniques for fixation recognition with respect to often, however, swift head movements or extreme position extracting data from noisy, jittery, error-filled stream and for changes were causing these issues. Ohm et al. [2014] tried to addressing the problem of "Midas touch” where people look at find where people look at, when navigating in a large scale an item without having the look “mean” something. Jacob and indoor environment, and what objects can assist them to find Karn [2003] presented a list of promising eye tracking metrics their ways. They conducted a user study and assessed the for data analysis: visual attractions of objects with an eye tracker. Their findings show that functional landmarks like doors and stairs are most • Gaze duration - cumulative duration and average spatial likely to be looked at. In our case we can use these landmarks location of a series of consecutive fixations within an area of as reliable points of interest that can be used for finding the interest. location of the visitor in the museum. Beugher et al. [2014] • Gaze rate – number of gazes per minute on each area of presented a novel method for the automatic analysis of mobile interest. eye-tracking data in natural environment for object • Number of fixation on each area of interest. recognition. The obtained results were satisfactory for most of • Number of fixation, overall. the objects. However, a large scale variance results in a lower • Scan path – sequence of fixations. detection rate (for objects which were looked at both from very far away and from close by.) • Number of involuntary and number of voluntary fixations (short fixations and long fixations should be defined well in Schrammel et al. [2011, 2014] studied attentional behavior of term of millisecond units). users on the move. They discussed the unique potential and challenges of using eye tracking in mobile settings and Using handheld devices as a multimedia guidebook in demonstrated the ability to use it to study the attention on museums has led to improvement in the museum visit advertising media in two different situations: within a digital experience. Researches have confirmed the hypothesis that a display in public transport and towards logos in a pedestrian portable computer with an interactive multimedia application shopping street as well as ideas about a general attention has the potential to enhance interpretation and to become a model based on eye gaze. Kiefer et al. [2014] also explored new tool for interpreting museum collections [Evans et al. the possibility of identifying users’ attention by eye tracking in 2005, Evans et al. 1999, Hsi 2003]. Studies about integration the setting of tourism – when a tourist gets bored looking at a of multimedia guidebooks with eye tracking have already been city panorama – this scenario may be of specific interest for us made in the context of museums and cultural heritage sites. as locations or objects that attracted more or less interest may Museum Guide 2.0 [Toyama et al. 2012] was presented as a be used to model user's interest and trigger further framework for delivering multimedia content for museum’s services/information later on. Nakano and Ishii (2010) studied visitors which runs on handheld device and uses the SMI the use of eye gaze as an indicator for user engagement, trying viewX eye tracker and object recognition techniques. The also to adapt it to individual users. Engagement may be used visitor can hear audio information when detecting an exhibit. as an indicator for interest and the ability to adapt engagement A users' study was conducted in a laboratory setting, but not in detection to individual users may enable us also to infer a real museum. We plan to extend this work by integrating an interest and build/adapt a user model using this information. eye tracker into real museum visitors' guide system and Furthermore, Ma et al. [2015] demonstrated an initial ability to experiment it is realistic setting. extract user models based on eye gaze of users viewing videos. Xu et al. [2008] also used eye gaze to infer user Brône et al. [2011] have implemented effective new methods preferences in the content of documents and videos by the for analyzing gaze data collected with eye-tracking device and users attention as inferred from gaze analysis (number of how to integrate it with object recognition algorithms. They fixations on word/image). presented a series of arguments why an object-based approach may provide a significant surplus, in terms of analytical As we have seen, there is a large body of work about precision. Specifically they discussed solutions in order to monitoring and analyzing users' eye gaze in general and also reduce the substantial cost of manual video annotation of gaze in cultural heritage setting. Moreover, the appearance of behavior, and have developed a series of proof-of-concept mobile eye trackers opens up new opportunities for research in case studies in different real world situations, each with its mobile scenarios. It was also demonstrated in several own challenges and requirements. We plan to use their lessons occasions that eye gaze may be useful in enhancing a user in our study. Pfeiffer et al. [2014] presented "EyeSee3D", model, as it may enable to identify users' attention (and where they combined geometric modelling with inexpensive interests). Considering mobile scenarios, when users also carry 3D marker tracking to align virtual proxies with the real-world smartphones - equipped with various sensors - implicit user objects. This allowed classifying fixations on objects of modeling can take place by integrating signals from various interest automatically while supporting a free movement of the sensors, including the new sensor of eye-gaze for better participant. During the analysis of the accuracy of the pose modeling the user and offering better personalized services. So estimation they found that the marker detection may fail from far sensors like GPS, compass, accelerometers and voice several reasons: First, sometimes the participant looked detectors were used in modeling users' context and interests, (see for instance [Dim & Kuflik. 2014]). When we mention experience. The system will be evaluated in user studies, the mobile scenarios, we refer to a large variety of different participants will be students from University of Haifa. The scenarios – pedestrians' scenario differs from jogging or study will be conducted in Hecht museum1, which is a small shopping or cultural heritage scenario. The tasks are different museum, located at the University of Haifa that has both an and users' attention is split differently. The cultural heritage archeological and art collections. The study will include an domain is an example where users have long term interests orientation about using the eye tracker and the mobile guide, that can be modeled and the model can be used and updated then taking a tour with the eye tracker and handheld device, during a museum visit by information collected implicitly multimedia content will be delivered by showing information from various sensors, including eye-gaze. In this sense, the on the screen or by listening to audio by earphones. Data will proposed research extends and aims at generalizing the work be collected as follows: The students will be interviewed and of Kardan and Conati [2013]. Still, even though a lot of asked about their visit experience, and will be asked to fill research effort was invested in monitoring, analyzing and questionnaires regarding general questions such as if it is the using eye gaze for inferring user interests, so far, little research first time that they have visited the museum, their gender and attention was paid to users gazing behavior "on the go". This age, and more. Visit logs will be collected and analyzed for scenario poses major challenges as it involves splitting later use, we can come to conclusions about the exhibit attention between several tasks at the same time – avoiding importance and where the visitors tend to look, the positioning obstacles, gathering information and paying attention to of the exhibits, and the time of the visits or explorations. The whatever seems relevant, for many reasons. study will compare the visit experience when using two different system versions – a conventional one and one with an 4. RESEARCH GOAL AND QUESTIONS integrated eye tracker, we will choose the work of [Kuflik et Our goal is to examine the potential of integrating the eye al. 2012] that was conducted in Hecht museum and which uses tracking technology with a mobile guide for a museum visit “light weight” proximity based indoor positioning sensors for and try to answer the question: How can the use of mobile location-awareness as a comparison system for examining the eye tracker enhance the museum visit experience? Our user experience. focus will be on developing a technique for location awareness based on eye gaze detection and image matching, and integrate 6. PRELIMINARY RESULTS it with a mobile museum visitor’s guide that provides It was important to examine the accuracy of eye gaze detection multimedia content to the visitor. For that we will design and when using the Pupil Dev mobile eye-tracker device. For that, develop a system that runs on handheld device and uses Pupil we have conducted several small-scale user studies onsite. Dev [Kassner et al. 2014] eye tracker for identifying objects of interest and delivering multimedia content to visitor in the 6.1 The Pupil eye tracker museum. Then we will evaluate the system in a user study in a Pupil eye tracker [Kassner et al. 2014] is an open source real museum to find out how the use of eye tracker integrated platform for pervasive eye tracking and gaze-based with a multimedia guide can enhance the museum visit interaction. It comprises a light-weight eye tracking headset experience. In our study, we have to consider different factors that includes high-resolution scene and eye cameras, an open and constraints that may affect the performance of the system, source software framework for mobile eye tracking, as well as such as the real environment lighting conditions which are a graphical user interface to playback and visualize video and different from laboratory conditions and can greatly affect the gaze data. The software and GUI are platform-independent process of object recognition. Another aspect may be the and include algorithms for real-time pupil detection and position of the exhibits relative to the eye tracker holder, since tracking, calibration, and accurate gaze estimation. Results of the eye tracker device is mounted as this is constrained by the a performance evaluation show that Pupil can provide an museum layout. While having many potential benefits, a average gaze estimation accuracy of 0.6 degree of visual angle mobile guide can also have some disadvantages [Lanir et al, (0.08 degree precision) with a processing pipeline latency of 2013]. It may focus the visitor’s attention on the mobile device only 0.045 seconds. rather than on the museum artifacts [Grinter et al, 2002]. We will also examine this behavior and try to review whether the use of eye tracker in mobile guide can increase the looking time at the exhibits. In addition, we will try to build a system that runs in various real environments with different factors and have the same constraints such as the light and the position constraints. 5. TOOLS AND METHODS A commercial mobile eye tracker will be integrated into a Figure 1. Pupil eye-tracker (http://pupil-labs.com/pupil) mobile museum visitors' guide system as a tool for location awareness, interest detection and focus of attention by using computer vision techniques. Our hypothesis is that the use of 1 http://mushecht.haifa.ac.il/ the eye tracker in mobile guides can enhance the visit 6.2 User study 1: Look at a grid cells Five students from the University of Haifa, without any visual disabilities participated in this study. They were asked to look at wall-mounted grid from a distance of 2 meters and track a finger (see figure 2). On every cell that the finger pointed at, they were asked to look at for approximately 3 seconds. Data was collected for determining the practical measurement accuracy. The results were as follows: on average, fixation detection rate was ~80% (most missed fixations were in the edges/corners – see table 1 for details about misses). In Figure 4. Gallery exhibition addition, average fixation point error rate, in terms of distance from the center of grids, was approximately 5 cm (exact error 6.3 User study 2: Look at an exhibit rate can be calculated using simple image processing In this study we examined the accuracy of the eye tracker in a techniques for detecting the green circle and applying mapping realistic setting. One participant (1.79m tall) was asked to look transform to the real word). at exhibits at the Hecht museum. Several exhibits where chosen with different factors and constraints (see figure 4, 5, and 6). The main constraint in this case is the distance from the exhibit since the visual range gets larger when the distance grows, and mainly we have to cover all the objects that we are interested in. Table 2 presents the objects height from the floor and the distance of the participant from the object. The next step was to examine fixations accuracy after making sure that the participant is standing in a proper distance. The participant was asked to look at different points in the exhibit/scene. In Figure 2. User study 1. The finger points at the grid where the gallery exhibits, the scan path has been set to be the four the participant were asked to look at. The green circle is a corners of the picture and finally the center of it. Regarding fixation point given from the eye tracker. The size of each the vitrine exhibits, for each jug one point at the center has cell is 20x20 cm. been defined Cell # 6 18 19 23 24 Missed 5 5 3 5 5 Table 1. Experiment details. During the study we ran into several practical problems. The Pupil Dev eye tracker that we are using is not fitted for every person. The device consists of two cameras, the first for delivering the scene and the second directed to the right eye Figure 5. Mounted backlighted images exhibition for detecting fixations. In some cases when the device is not fitted correctly, the vision range got smaller and parts of the It’s important to note that the heights/distances relation is for pupil got out from the capture frame (see figure 3 for visual range (having the objects in the frame of the camera) example). As a consequence no fixations were detected. and not for fixations detections. Since missed fixations could Another limitation was that when using the eye tracker with be as a result of a set of constraints and not the distance from tall participants, they have to step back from the object which the object, thing that we have not examined yet. negatively affects the accuracy. Figure 3. Screen capture from eye camera. Figure 6. Vitrine backlighted exhibition. Exhibit width height Height from Stand type (cm) (cm) floor (cm) distance (cm) Vitrine 80 25 150 150 Exhibit width height Height from Stand 8.2 Object matching type (cm) (cm) floor (cm) distance (cm) The matching procedure will be done in three steps: shelf 80 15 120 230 1. Eye-tracker scene camera frame is taken (figure 7) and 80 20 90 310 image-to-image matching applied. The result is an image 80 15 40 390 with labeled regions in the current scene’s frame (figure 8). Gallery 60 67 150 200 Table 2. Experiment details – we considered the three most left 2. Mapping transformation – We need to transform the shelves in the vitrine exhibit shown in figure 6. fixation point in the eye-tracker scene camera to a suitable/matched point in the image that we got in step one (image from the data-set with labeled regions), since the 7. SYSTEM DESIGN viewpoint of the objects can be different from this in the A smart context-aware mobile museum visitors' guide may data set. For example one image is rotated relative to the provide the visitor with personalized relevant information other or one is zoomed in/out as a result of standing in from the vast amount of content available at the museum, different distance from the object when the data-set image adapted for his or her personal needs. Furthermore, the system was taken. may provide recommendations and location-relevant information. However, the potential benefit may also have a 3. Finding the object - This is step is simple since we have a cost, the notifications may interrupt the user in the current task mapped fixation points and labeled regions. What remains and be annoying in the wrong context. Beja et al. [2015] is determining for which object the point does it relates (or examined the effect of notifications in a special leisure it relates to nothing). scenario - a museum visit. Following Beja et al [2015], we will consider three different scenarios: I. The Visitor is looking at an exhibit. The region of interest will be defined as the region from the scene around the gaze fixation point. Then object matching procedure will be applied (see section 8). It will enable us to determine both the visitor’s position and the object of interest. II. The visitor is looking at the tablet. This could be done in two ways: 1) the visitor is watching multimedia information, in this scenario there is nothing to do for him. Figure 7. Example of eye-tracker scene camera. The green 2) The visitor may need service from the system or a point is the fixation point. recommendation, so it is the right time to deliver him. III. The visitor is wandering in the museum. According to Beja 9. DISCUSSION et al. [2015], it is the best time for sending notifications. We conducted these small-scale user studies in order to gain initial first-hand experience with the eye-tracker in a realistic As a basic system we will use the PIL museum visitor's guide setting. Furthermore, we tried to clarify which exhibits are system [Kuflik et al 2012; Lanir et al. 2013]. The system is a appropriate to be included in our future study and, given the context aware, mobile museum visitors' guide system. Its limitation of the device, what portion of the museum exhibits positioning mechanism is based on proximity based RF may be included in general. Not surprisingly, we got 100% technology that enables to identify the visitor's position – accuracy rate when we examined the device in the art wing when the visitor is near a point of interest. As vision is the since all the pictures are placed in ideal height. Regarding the main sense for gathering information, we plan to replace the archeological wing, it is considerably more challenging system's positioning component with an eye-tracker based environment, since objects are placed in different heights and positioning and object of interest identification component. have unequal sizes. As a result the visitor may have to stand Hence we will enhance the positioning system by providing far away from the objects in order to get them into the eye- the system the ability to pin-point the object of interest. The tracker front camera frame, a fact that can negatively affect the rest of the system will remain unchanged. Having these two visit experience. In the case of archeological wing we versions of systems will enable us to compare and evaluate the approximate that about 60% of the exhibits may be detectable benefits of the eye-tracker as a positioning and pointing device with the current device. Regarding the low-height exhibits we in the museum. don’t know yet whether they can be considered or not. More challenging exhibits are these that are placed in harsh light 8. OBJECT MATCHING PROCEDURE conditions or placed in low height (see figure 9 for example) 8.1 Data-set preparation and/or these that are too large to fit in one frame (see figure 10 A set of images of the exhibits will be taken, each image may for example). contain one or more objects. Each image will be given a distinct label value and size of region around the object (in terms of width and height – rectangular shape). earphones or by watching slides. Furthermore, knowing exactly where the visitor look in the scene (specific object) will allow us to deliver personalized information. Our research will be a supplement to the nowadays mobile museum guide that uses location-awareness technology and techniques that enhances the visit experience. The system can also be extended and used in other venues such as outdoors cultural heritage sites as well as shopping centers/markets after further validation. REFERENCES Figure 8. Image-to-image matching. The yellow rectangles are the regions around each object. The green point is the [1] Ardissono, L., Kuflik, T., & Petrelli, D. (2012). fixation point after performing proposed mapping Personalization in cultural heritage: the road travelled and transformation. The corresponding region would be R3. the one ahead. User modeling and user-adapted interaction, 22(1-2), 73-99. [2] Barrow, H. G., Tenenbaum, J. M., Bolles, R. C., & Wolf, H. C. (1977). Parametric correspondence and chamfer matching: Two new techniques for image matching (No. TN-153). SRI international, Menlo Park CA, Artificial Intelligence center. [3] Beja, I., Lanir, J. and Kuflik, T. Examining Factors Influencing the Disruptiveness of Notifications in a Mobile Museum Context. Human–Computer Interaction just-accepted (2015). [4] Bitgood, S. (2002). Environmental psychology in museums, zoos, and other exhibition centers. Handbook Figure 9. A challenging exhibition: harsh light conditions of environmental psychology, 461-480. and low-height. [5] Brône, G., Oben, B., Van Beeck, K. Goedemé T. (2011). Towards a more effective method for analyzing mobile eye-tracking data: integrating gaze data with object recognition algorithms. UbiComp ‘11, Sep 17–Sep 21, 2011, Beijing, China. [6] Bulling, A., Dachselt, R., Duchowski, A., Jacob, R., Stellmach, S., & Sundstedt, V. (2012). Gaze interaction in the post-WIMP world. In CHI'12 Extended Abstracts on Human Factors in Computing Systems, 1221-1224. [7] Bulling, A., & Zander, T. O. (2014). Cognition-aware computing. Pervasive Computing, IEEE, 13(3), 80-83. [8] De Beugher, S., Brône, G., & Goedemé, T. (2014). Automatic analysis of in-the-wild mobile eye-tracking Figure 10. A challenging exhibit: too big to fit in one frame. experiments using object, face and person detection. In Proceedings of VISIGRAPP 2014, 1, 625-633. 10. CONCLUSIONS AND FUTURE WORK [9] Dim, E., & Kuflik, T. (2014). Automatic detection of This paper presents a work-in-progress that aims at exploring social behavior of museum visitor pairs. ACM the potential contribution of the mobile eye tracking Transactions on Interactive Intelligent Systems, 4(4), 17. technology in enhancing the museum visit experience. For that [10] Evans, J., Sterry, P. Portable computers & interactive we have done small-scale experiments in order to get an media: A new paradigm for interpreting museum understanding of the performance of the system in realistic collections. In: Bearman, D., Trant, J. (eds.) Cultural setting. We got satisfactory results from these studies and an understanding of the limitations of the equipment. The next Heritage Informatics 1999: Selected papers from ICHIM step in the study is to design and build a museum mobile guide 1999, 93–101. Kluwer Academic Publishers, Dordrecht. that extends the use of mobile eye tracking as a tool for [11] Falk, John H., and Lynn D. Dierking. Learning from identifying the visitor position and points of interests. We will museums: Visitor experiences and the making of use the eye-tracker scene camera captures and the collected meaning. Altamira Press, 2000. gaze data to develop a technique for location-awareness. The [12] Fitts, P. M., Jones, R. E., & Milton, J. L. (1950). Eye system will run on tablet, and the multimedia content will be movements of aircraft pilots during instrument-landing delivered to the participants by listening to audio guide via approaches. Aeronautical Eng. Review 9(2), 24–29. [13] Goferman, S., Zelnik-Manor, L., & Tal, A. (2012). [27] Ma, K. T., Xu, Q., Li, L., Sim, T., Kankanhalli, M., & Context-aware saliency detection. Pattern Analysis and Lim, R. (2015). Eye-2-I: Eye-tracking for just-in-time Machine Intelligence, IEEE Transactions on, 34 (10), implicit user profiling. arXiv preprint arXiv:1507.04441. 1915-1926. [28] Nakano, Y. I., & Ishii, R. (2010). Estimating user's [14] Grinter, R. E., Aoki, P. M., Szymanski, M. H., Thornton, engagement from eye-gaze behaviors in human-agent J. D., Woodruff, A., & Hurst, A. (2002). Revisiting the conversations. In Proceedings of IUI 2010.139-148. visit: understanding how technology can shape the [29] Naphade, M. R., Yeung, M. M., & Yeo, B. L. (1999). museum visit. In Proceedings of CSCW 2002,146-155. Novel scheme for fast and efficient video sequence [15] Gruen, A. (1985). Adaptive least squares correlation: a matching using compact signatures. Electronic Imaging. powerful image matching technique. South African 564-572. International Society for Optics and Photonics. Journal of Photogrammetry, Remote Sensing and [30] O. Stock, M. Zancanaro, F. Pianesi, D. Tomasini, and C. Cartography, 14(3), 175-187. Rocchi. 2009. Formative evaluation of a tabletop Display [16] Hampapur, A., Hyun, K., & Bolle, R. M. (2001, meant to orient casual conversation. Journal of December). Comparison of sequence matching techniques Knowledge, Technology and Policy 22(1), 17–23. for video copy detection. In Electronic Imaging 2002 . [31] Ohm, C., Müller, M., Ludwig, B., & Bienk, S. (2014). 194-201. International Society for Optics and Photonics. Where is the Landmark? Eye Tracking Studies in Large- [17] Hendrickson, K., & Ailawadi, K. L. (2014). Six lessons Scale Indoor Environments. for in-store marketing from six years of mobile eye- [32] Packer, Jan, and Roy Ballantyne. (2005) "Solitary vs. tracking research. Shopper Marketing and the Role of In- shared: Exploring the social dimension of museum Store Marketing. Review of Marketing Research, 11, 57- learning." Curator: The Museum Journal 48.2 177-192. 74. [33] Pinto, Nicolas, David D. Cox, and James J. DiCarlo. [18] Kardan, S., & Conati, C. (2013). Comparing and "Why is real-world visual object recognition hard?." Combining Eye Gaze and Interface Actions for (2008): e27. Determining User Learning with an Interactive [34] Pfeiffer, T., & Renner, P. (2014). Eyesee3d: A low-cost Simulation. In proceedings of UMAP 2013, 215-227. approach for analyzing mobile 3d eye tracking data using [19] Kassner, M., Patera, W., & Bulling, A. (2014, computer vision and augmented reality technology. September). Pupil: an open source platform for pervasive In Proceedings of the Symposium on Eye Tracking eye tracking and mobile gaze-based interaction. Research and Applications. 369-376. ACM. In Proceedings of the 2014 ACM International Joint [35] ROBERT J. K. JACOB (1991). The Use of Eye Conference on Pervasive and Ubiquitous Computing: Movements in Human-Computer Interaction Techniques: Adjunct Publication. 1151-1160. ACM. What You Look At is What You Get, ACM Transactions [20] Katy Micha, Daphne Economou (2005). Using Personal on Information Systems, 9(3), April 1991, 152-169. Digital Assistants (PDAs) to Enhance the Museum Visit [36] Robert J. K. Jacob, Keith S. Karn (2003). Eye Tracking in Experience. In proceedings of PCI 2005, Volas, Greece, Human–Computer Interaction and Usability Research: November 11-13, 2005. Proceedings. 188-198. Ready to Deliver the Promises, Elsevier Science BV. [21] Kiefer, P., Giannopoulos, I., Kremer, D., Schlieder, C., & [37] Schrammel, J., Mattheiss, E., Döbelt, S., Paletta, L., Raubal, M. (2014). Starting to get bored: An outdoor eye Almer, A., & Tscheligi, M. (2011). Attentional behavior tracking study of tourists exploring a city panorama. of users on the move towards pervasive advertising In Proceedings of the Symposium on Eye Tracking media. In Pervasive Advertising, 287-307.. Research and Applications (pp. 315-318). ACM. [38] Schrammel, J., Regal, G., & Tscheligi, M. (2014). [22] Kim, C., & Vasudev, B. (2005). Spatiotemporal sequence Attention approximation of mobile users towards their matching for efficient video copy detection. Circuits and environment. In CHI'14 Extended Abstracts on Human Systems for Video Technology, IEEE Transactions Factors in Computing Systems, 1723-1728. on, 15(1), 127-132. [39] S.Hsi (2003). A study of user experiences mediated by [23] Kuflik, T., Lanir, J., Dim, E., Wecker, A., Corra, M., nomadic web content in a museum. The Exploratorium, Zancanaro, M., & Stock, O. (2012). Indoor positioning in 3601 Lyon Street, San Francisco, CA 94123 cultural heritage: Challenges and a solution. In Electrical [40] Takumi Toyama, Thomas Kieninger, Faisal Shafait, & Electronics Engineers in Israel (IEEEI), 2012 IEEE 27th Andreas Dengel. Gaze guided object recognition using a Convention of. 1-5. IEEE. head-mounted eye tracker. ETRA '12 Proceedings of the [24] Lanir, J., Kuflik, T., Dim, E., Wecker, A. J., & Stock, O. Symposium on Eye Tracking Research and Applications, (2013). The influence of a location-aware mobile guide on 91-98, ACM New York, NY, USA 2012 museum visitors' behavior. Interacting with Computers, [41] Thirion, J. P. (1998). Image matching as a diffusion 25(6), 443-460. process: an analogy with Maxwell's demons. Medical [25] Laurent I.(2007) Visual salience. Scholarpedia, 2(9):3327. image analysis, 2(3), 243-260. [26] Leinhardt, G., and Knutson K. Listening in on museum [42] Xu, S., Jiang, H., & Lau, F. (2008). Personalized online conversations. Rowman Altamira, 2004. document, image and video recommendation via commodity eye-tracking. In Proceedings of RecSys 2008 [43] Yalowitz, S.S. and Bronnenkant, K. (2009) Timing and [44] Zhang, Z., Deriche, R., Faugeras, O., & Luong, Q. T. tracking: unlocking visitor behavior. Visit. Stud., 12, 47– (1995). A robust technique for matching two uncalibrated 64. images through the recovery of the unknown epipolar geometry. Artificial intelligence, 78(1), 87-119.