The Madeira Touch: Encouraging Visual-Spatial Exploration using a Tactile Interactive Display Catia Prandi Abstract ARDITI, Madeira-ITI The current information marketplace for tourists is Funchal 9020-105, Portugal dominated by for-profit purveyors of information. catia.prandi@m-iti.org Potential visitors must rely on experts-for-hire or search engine results in order to learn about a desired Catherine Chiodo destination. In this paper, we introduce The Madeira Ricjeareu Villaflor Touch, a multimodal display installation rooted in the Carnegie Mellon University unique characteristics of Madeira, which allows users to Pittsburgh PA 15213, USA explore the island by selecting a type of scenery and catherine@chiodo.org showing the user-generated photos of that type of rj.villaflor@gmail.com scenery in a map-based interface. To make this pervasive display more engaging, we designed an Nicolas Autzen exploratory tactile-input mode of interaction: users will Johannes Schöning be able to touch a physical object, representing a type University of Bremen of scenery (a rock for mountains, a seashell for the sea, Bremen 28359, Germany etc.), which will then bring up suitable photos of that n.autzen@uni-bremen.de type of scenery overlaid on a map of the island. The schoening@uni-bremen.de display will help users to form their mental image of the island and to plan trips that best suit their interests. Author Keywords Pervasive display, tactile interaction, multimodal interaction, user-generated content, digital signage ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., Copyright is held by the author/owner(s). HCI): Miscellaneous. CHItaly ’17, September 18-20, 2017, Cagliari, Italy. Introduction & Motivation The Madeira Touch, a pervasive display that allows Scenarios of Use In an increasingly connected world, travelers seeking users to correlate photos with locations, using a Scenario 1: While waiting at local experiences may encounter a paradox. While it is multimodal interaction that allow users to select either the tourism office, a visitor easy to find information about destinations and places a type of scenery (touching a physical object) or a notices a display with boxes that they may wish to visit, it is also increasingly location (using the map-based visualization). Madeira containing different objects difficult to differentiate between the many options represents an ideal place to develop a new way for surrounding it, in front of the available. At the same time, travelers who have visitors to explore the island, however, it is our entrance. She touches one witnessed the homogenizing effects of globalization intention for this system to be adaptable to other object and notices the display may place a particular premium on unique experiences tourist destinations, levering on the location’s unique showing images from a that can only be found in certain places. offerings and characteristics. levada walk in Madeira. She removes her hand from the In this project, we took the island of Madeira as a object and the display returns representative case study of a well-known touristic to the map of the island. She destination that could be made more discoverable to continues to touch the the island’s visitors. However, while tourists may objects and see the images choose to visit Madeira because of its beautiful scenery associated with them. She and outdoor activities, often related to the island’s leaves the display with a levadas, remote canals that serve as walking paths, better understanding of the they may not know what destinations on the island are opportunities on the island. best suited to the kinds of sceneries they hope to see. Currently, travelers who hope to experience natural Scenario 2: A local Madeiran beauty can plan their trips by either starting with a visits the tourism office to possible location and attempting to find correlating see the new interactive photos, or by beginning with photos of destinations display. While interacting they would like to visit and then attempting to find Figure 1: The Madeira Touch in context with it, he notices that one of location information. Both approaches suffer the photos is an image he breakdowns when photos are not tagged with the The Madeira Touch took on Instagram. He feels commonly-used location names. Our solution is to provide the Madeira’s main tourism gratified that one of his office, with our pervasive display. There, visitors to the images is contributing to the In the field of pervasive display systems [4], we island will be able to explore a map of Madeira using experience available on the designed our solution to investigate how the use of an two modes of interaction: 1. traditional map-based display and to the information exploratory tactile mode of interaction to provide touchscreen interaction (Figure 2), or 2. exploratory provided to tourists on the georeferenced visual information in the form of user- tactile-input interaction (Figure 3). With the first type of island. generated photos of points of interest (POIs) can interaction, visitors will be able to touch the digital map facilitate visitors in discovering locations on the touristic and see user-generated photos of that location. With destination, enhancing their experience. The result is the second one, users will be able to touch a physical object that corresponds to a type of scenery on the interaction with the data, engendering an experience Scenario 3: While his island, which will then bring up user-generated photos that is less goal-oriented and more exploratory. On the parents are in the queue, a of that type of scenery (Figure 1). By enabling both other hand, the use of tactile physical objects can raise 15-year-old notices a monitor forms of interaction, we intend to encourage the ‘affordance blindness’ issue, defined as the inability surrounded by different exploratory scenery discovery as well as practical trip- to understand the interaction modalities of a public natural objects in display planning in context in which multiple users, such as display [3]. Our solution aims to moderate this boxes, in front of the family groups, can co-experience the display [5]. problematic providing visual hints to attract the user’s entrance. He tries to touch a attention (as described in Scenario 3). location on the monitor, Design Concepts expecting a touch-based Traditional tourism information relies on professionally- interaction with the system. produced content which is limited both in quantity and In response to his touch, the in coverage. We intend for this system to serve as an display shows a photo of that exploration of how UGC from social media can be area. At the same time, a box curated to provide dynamic, updated and custom sets lights up, grabbing his of information for specific audiences (i.e., visitors). attention. He notices that the box holds a tree branch and While interactive displays have a high potential to Figure 2: Users can touch the thumbnails on the screen. that the photo on the screen engage passersby, they frequently go unnoticed and is full of trees. Intrigued, he unused [7, 11], confirming the so called ‘display decides to touch the tree blindness’ effect [8]. By situating our display in a branch and the monitor strategic location (i.e., in front of the entrance) of a begins to display photos of tourism office where visitors often wait to speak to forests in different areas of someone, we intend to mitigate this issue, exposing the island. He touches visitors to the display at a time when they will be another object, then another. inclined to investigate. To further increase visitor Figure 3: Users can directly touch the physical objects. When his parents are done, engagement, we have incorporated a novel form of he brings them over to the input in the form of physical objects that a user may Implementation display and together they touch to experience certain kinds of sceneries. In fact, The Madeira Touch software architecture is composed explore the island. studies of initial engagement with interactive displays of three main modules (as shown in Figure 4). The have found that physical interactions prompt greater Geotagged Photos Retrieval Module collects geotagged rates of engagement among passersby [6]. Moreover, photos and paths related to the main touristic module our solution aims to overcome the ‘interaction first retrieves and stores georeferenced routes by blindness’ [10] that often plagues public displays by pedestrian walks that characterize the island. This providing users with novel and suggestive physical stage, we have decided to use OpenStreetMap (OSM), objects that encourage non-linear exploration. This an open source system that allows users to voluntarily paradigm encourages a very different kind of collect and share GPS tracks and georeferenced data (i.e. Point of Interests). In Madeira, an island with an the object the user touches, showing the information in area of 802 km², the OSM dataset includes 16000 a Google Maps based interface. Regarding the hardware points and 24000 lines along with walking paths requirements, The Madeira Touch utilizes a touch (levadas). There are also private datasets gathered by screen monitor and sensors to indicate when an object companies that have collected GPS tracks specifically has been touched. Considering the design of our related to walking paths such as Walk Me Madeira system, simple motion sensors should work well. The (http://www.walkmeguide.com/). Based on this dataset current design also uses LED lights in each box to of GPS tracks and routes, the module retrieves public emphasize the way in which each object corresponds to geotagged photos from different social media platforms a certain type of photo. This way, even when the user and photo blogs to continually integrate UGC (i.e. is interacting directly with the touchscreen, the relevant photos) to enrich the user experience of our system object will light up, indicating the relationship between [12]. At this stage, the system includes the Instagram object and image. Figure 4: System architecture and Flickr platforms, both of which provide developers with APIs for retrieving public photos based on Conclusion and Future Work locations and/or tags. Our concept combines three characteristics in a unique way to make an engaging pervasive display for visitors The Classification Module is the core of our system, to the island of Madeira. By using images from social because it enables (i) the elimination of photos which media, the system insures that the content remains include faces and other non-nature showing images and dynamic and accurate. By allowing users to explore the (ii) the categorization of the collected pictures based on data, either by location or by scenery type, the system the main objects (rocks, sand, etc.) and scenery (cliff, allows for scenery exploration and practical trip forest, etc.) in each image. Each of these categories is planning. Finally, by creating a tactile form of correlated with a physical object with which the users interaction, the system encourages users to consider can interact. Different kinds of machine learning the natural materials represented, allowing them to algorithms have been developed for the recognition and form a more sensual and complete mental picture of classification of faces/landscapes/nature elements the destination. While all three of these characteristics represented in photos from social media platforms and are transferable and could be applied to other photo blogs [1, 2, 9, 13]. However, this system could destinations, it is the authors’ belief that, for future also use a crowdsourcing approach to let participating installations of this solution, the appeal and utility of users manually check the photos. A third approach such a system relies on its ability to accurately reflect would be a system like the Google ReCAPTCHA, which the unique character of the local environment. To asks users to solve a puzzle by selecting all the images evaluate the effectiveness of our pervasive display in that represent a specific element. engaging users and enhancing their visiting experience, overcoming the display/interaction/affordance blindness The Visualization Module is the final step in this system issues, we plan to install the system in the Madeira’s and connects the categorized geotagged photos with main tourism office. References Pervasive Displays (PerDis '13), 19-24. 1. Nuttapoom Amornpashara, Yutaka Arakawa, http://doi.acm.org/10.1145/2491568.2491573 Morihiko Tamai and Keiichi Yasumoto. 2015. 8. Jörg Müller, Dennis Wilmsmann, Juliane Exeler, Landscape photo classification mechanism for Markus Buzeck, Albrecht Schmidt, Tim Jay, and context-aware photography support system, In Antonio Krüger. 2009. Display blindness: The effect Proceedings of Conference on Consumer Electronics of expectations on attention towards digital (ICCE 2015), 663-666. signage. Pervasive Computing (2009): 1-8. http://doi.acm.org/10.1109/ICCE.2015.7066570 http://dx.doi.org/10.1007/978-3-642-01516-8 2. Pu Cheng and Jie Zhou. 2011. Automatic Season 9. Mor Naaman, Susumu Harada, QianYing Wang, Classification of Outdoor Photos, In Proceedings of Hector Garcia-Molina, and Andreas Paepcke. 2004. the Conference on Intelligent Human-Machine Context data in geo-referenced digital photo Systems and Cybernetics, 46-49. collections. In Proceedings of the conference on http://doi.acm.org/10.1109/IHMSC.2011.18 Multimedia (MULTIMEDIA '04), 196-203. 3. Jorgos Coenen, Sandy Claes, and Andrew Vande http://dx.doi.org/10.1145/1027527.1027573 Moere. 2017. The concurrent use of touch and mid- 10. Gonzalo Parra, Joris Klerkx, and Erik Duval. 2014. air gestures or floor mat interaction on a public Understanding Engagement with Interactive Public display. In Proceedings of the Symposium on Displays: an Awareness Campaign in the Wild. In Pervasive Displays (PerDis '17). Article 9, 9 pages. Proceedings of the Symposium on Pervasive DOI: https://doi.org/10.1145/3078810.3078819 Displays (PerDis '14), 180-186. 4. Nigel Davies, Sarah Clinch, and Florian Alt. 2014. http://dx.doi.org/10.1145/2611009.2611020 Pervasive displays: understanding the future of 11. Peter Peltonen, Esko Kurvinen, Antti Salovaara, digital signage. Synthesis Lectures on Mobile and Giulio Jacucci, Tommi Ilmonen, John Evans, Antti Pervasive Computing 8.1 (2014): 1-128. Oulasvirta, Petri Saarikko. 2008. It’s mine, don't 5. Jodi Forlizzi and Katja Battarbee. 2004. touch!: interactions at a large multi-touch display Understanding experience in interactive systems. in a city centre. In Proceedings of the Conference In Proceedings of the conference on Designing on Human Factors in Computing Systems (CHI interactive systems: processes, practices, methods, '08), 1285–1294. and techniques (DIS '04), 261-268. http://dx.doi.org/10.1145/1357054.1357255 http://doi.acm.org/10.1145/1013115.1013152 12. Pavel Serdyukov, Vanessa Murdock, and Roelof van 6. Wendy Ju and David Sirkin. 2010. Animate objects: Zwol. 2009. Placing flickr photos on a map. how physical motion encourages public interaction. In Proceedings of the conference on Research and In Proceedings of the conference on Persuasive development in information retrieval (SIGIR '09), Technology (PERSUASIVE'10), 40-51. 484-491. http://doi.acm.org/10.1007/978-3-642-13226-1_6 http://doi.acm.org/10.1145/1571941.1572025 7. Kazjon Grace, Rainer Wasinger, Christopher Ackad, 13. Feng Tang, Daniel R. Tretter and Chris Willis. 2011. Anthony Collins, Oliver Dawson, Richard Gluga, Event classification for personal photo collections. Judy Kay, and Martin Tomitsch. 2013. Conveying In Proceedings of Conference on Acoustics, Speech interactivity at an interactive public information and Signal Processing (ICASSP), 877-880. display. In Proceedings of the Symposium on http://doi.acm.org/10.1109/ICASSP.2011.5946544