Technological and Methodological Tools for Personalized Touchless Applications Matteo Valoriani Politecnico di Milano, Dipartimento di Elettronica, Informazione e Bioingegneria, Milano, Italy valoriani@elet.polimi.it Abstract. The objective of the research is to build technological and methodo- logical solutions to support the development of interactive applications that ena- ble a fluid interaction with multimedia contents. The objective is pursued em- ploying different paradigms of content presentation, interaction and control, sup- porting individual and social tasks, and through customization with respect to the characteristics of the intended users and situations of use. The case studies for understanding user requirements and for applying the research results, are envi- sioned in the domain of cultural heritage, tourism and healthcare. Keywords. Multimedia Systems, User Interfaces, Touchless Interaction 1 Introduction The recent evolution of ICT allows for the creation of innovative User Experiences (UXs) that, at individual and collective level, expand our access to and interaction with digital information in a way that would have been unconceivable just few years ago. In this arena, the research will address the design and development of novel interac- tive applications placed at the intersection of three macro categories: mobile [2], mul- titouch [3], and touchless [4]. Applications in the mobile category are those delivered on hand-held devices, for instance smartphones or tablets, which make contents and services available anywhere- anytime. In the second category there are applications involving multi-touch large sur- faces (as Microsoft Pixelsense[5] or PQLabs Multi-Touch Wall[6]), where multimedia content presentation can benefit from large scale display, and multi-user synchronous and asynchronous interaction is supported in a fluid and natural manner. This project explores problems and challenges arising from a new kind of interac- tion: touchless interaction. This category, probably the most heterogeneous one, com- prises applications developed in such a way that user interaction does not require phys- ical contact with input/output devices, but is intangible, and exploits technologies such as voice, gesture, or body movement recognition (e.g., Wii[7][8] and Kinect [9]). The ultimate objective is to build technological and methodological solutions to sup- port the development of interactive applications that enable a fluid interaction with multimedia content, support individual and social tasks, and facilitate the customization with respect to the characteristics of the intended users and situations of use. Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. In order to include the research in tangible context of use, two areas will be ad- dressed: Cultural heritage/tourism and Healthcare and disabilities support. 1.1 Cultural heritage/tourism This domain involves a variety of user profiles, needs, situations of use, and a large amount of information (think of museums, libraries, indoor or outdoor exhibitions, monuments, archeological sites, …). This is a particularly rich and stimulating field where the integration of different contents, devices and interaction paradigms, com- bined with personalization and contextualization features, is particularly promising with respect to the creation of more compelling and unique experiences for end users [10][11]. 1.2 Healthcare and disabilities support Limited research in this arena has explored the potential of “motion-based” (or “full- body”) touchless interaction. This paradigm exploits sensing devices, which capture, track and decipher body movements and gestures without users needing to wear addi- tional devices (e.g., data gloves, head mounted display, remote controllers, or body markers). Several authors claim that motion-based touchless interaction has the poten- tial to be more ergonomic and “natural” than other forms of interaction [12][13][14]. The gestural dimension resembles one of the most primary forms of human-to-human communication – body expression; body involvement can enhance engagement: the “come as you are” feature removes the burden of physical contact with technology, making the user experience more pleasant. 2 Research Challenges The work must face a number of research challenges. First of all, there is a Techno- logical Challenge, since: 1. Different users perform gesture in different ways; 2. Context changes the meaning of users’ gesture; [1] 3. Different hardware that allow touchless interaction will be available in the future. A second important issue to be addressed is the UX (User Experience) Design Chal- lenge. Gestures cannot be viewed as standard and the design of gesture is fundamental in order to create a “natural” user experience. From this point of view, the critical requirements are related to personalization, contextualization, and natural interaction of applications. A UX is more engaging and effective if multimedia contents and in- teraction mechanisms are customized with respect to the actual user(s) profile and her needs, and when these elements are appropriate for the actual “context” (e.g., the user’s physical disease, the situation where the UX takes place, the characteristics of the Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. physical location, and the social circumstances - individual vs. multi-user). Mastering all these aspects increases the complexity of the UX design space. It is therefore important to identify a conceptual modeling framework, which plays the same role as conceptual models in other areas, such as web applications. A model- ing framework identifies the dimensions along which the multi-device/multi-paradigm personalized/ contextualized UXs can be designed providing designers and developers with a “language” to specify the characteristics of the applications under design, as well as guidelines to organize their work; in addition, it offers a taxonomy for the quality variables to be measured while evaluating the final UXs. Mobile Single Multitouch In group Touchless Technology UX … App. Modeling Cultural heritage/ domain User tourism Situation Disabilities support Fig. 1. The research space at a glance 3 Research methodology and work plan The development of the proposed research is organized into a set of macro-tasks (some of which running in parallel) defined as follow. Analysis of the current state of the art in academic and industry contexts, with respect to: i) existing technologies and available interaction paradigms; ii) existing applications in the Cultural heritage/tourism and Healthcare and disabil- ities support. Contextual Studies [12] in cultural heritage environments (museums) to understand the profile of potential users, to unveil their needs with respect to cultural contents, and to identify patterns of (social or individual) behavior during their visits. Definition of the user requirements (in terms of contents, interaction, personaliza- tion, and contextualization needs) for multi-device coordinated applications in the cul- tural heritage domain. Definition of the technological requirements for an integrated platform supporting application execution. Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. Design, prototyping, and implementation of a middleware that can support gesture recognition. Design and prototyping of a sample (set of) application(s) that exploits the mid- dleware’s capabilities and offers case studies for its technical tuning and testing. User testing of such application(s), first in a controlled laboratory environment, and them in a real context (e.g., a real museum or therapeutic centers). Based on the experience gained in the previous tasks, definition of a conceptual framework (comprising guidelines, modeling concepts and primitives) to support the design and development of the considered class of applications. 4 Technological Framework Currently the writer is developing a framework for identifying and interpreting the users’ gestures inserting also fuzzy rules in order to decrease the dependence on param- eters and make the system more flexible. I intend to try to apply some algorithms of ANN to modify dynamically the weights of my network. There are two main compo- nents in the framework: the GestureRecognizer and the GestureManager (Figure 2). Fig. 2. Gesture Recognition Framework The GestureRecognizer dynamically analyses a subset of all body skeleton data de- tected by the sensing device (considering only information related to torso and upwards joints) and progressively checks if such data correspond to a gesture included in the set of the possible ones. A 2D active zone, a start condition, a set of constraints, and an end condition model each gesture. For example, a user “horizontal swipe of the right hand” (Figure 3) movement is interpreted as “Horizontal Browsing” if: 1. the right hand is detected in the area identified by the horizontal line “from left shoulder to twice the distance among shoulders” and the vertical line from “from left shoulder to the center of mass (belly) line” (active zone); 2. the right hand is moving horizontally (start condition); 3. the motion continues with minimum speed V (constraint); 4. the traversed distance is > D (end condition). Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. Fig. 3. Gesture schema The GestureRecognizer implements small modules to recognize some generic char- acteristics of movement. Composing these modules as a network, it is possible to model and recognize many gestures reusing the same code. The GestureRecognizer is fully parametric so that the designers can easily change the setting of gesture specification parameters to adapt them to the anthropometric char- acteristics of a specific user. Once a gesture has been recognized, its data are transmitted to the GestureManager for interpretation and execution. The semantics of a gesture (i.e., its effect on the appli- cation state) depend on the current execution context. The GestureManager checks for gesture intentionality and it executes the corresponding application state. 5 Case Studies This section includes a fast overview of a selection the projects developed in the context of this research in order to provide a better understanding of the different as- pects of the research. 5.1 Small screen: MOTIC Touchless gestural interaction in-the-small has received so far only a marginal atten- tion in industry and research. Still, this paradigm has a potential in a number of inter- esting domains. In addition, it is challenging from a research point of view. Building any touchless gesture-based application has an intrinsic complexity related to both achieving accurate and meaningful gesture recognition and identifying natural, intuitive and meaningful gesture vocabularies appropriate for the tasks under consideration [16]. When dealing with touchless gestural interaction in the small, this complexity is ex- acerbated by the fact that gestures and their detection must be designed in relation to visual interfaces that face a number of constraints: i) size; Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. ii) number of interface elements and contents that can be displayed, and their dimension; iii) digital space available to provide feedbacks to user’s interaction. MOTIC (MOtion-based Touchless Interactive Cooking system) is a case study in which I have applied touchless gestural interaction in the small to the domain of house- hold appliances. MOTIC comprises a conventional oven that is integrated with a small display and a Kinect motion sensing device (Figure 4) to enable users control cooking behavior thought body motion and touchless gestures. Fig. 4. Front view of MOTIC with the integrated small display and Kinect sensing device The design of any system based on gestural interaction involves two main dimen- sions. One concerns the definition of the gesture language, which specifies the shape (trajectory) and dynamics of body movements that express the user’s intentions, and communicate to a system while performing the tasks identified during requirement elic- itation. The second design dimension is related to the specification of the visual inter- face on display, presenting information and functions that the user can operate on and feedbacks on interaction. Finally, in applications requiring a smooth integration of the gestural system with the physical environment, a third design dimension needs to be addressed, concerning the actual positioning of hardware devices (sensing device and display). This dimension, in the case of MOTIC, is related the actual positioning of the Kinect within the home kitchen furniture (being the display position fixed). To address the three design dimensions, we have followed an iterative user-centered design process during which we have repeatedly defined, implemented, and tested ges- tures, visual interfaces, and Kinect positions, using progressively more sophisticated artifacts (from paper based mockups to functional prototypes) and involving users along the entire process. In my research this has been very important to understand two aspect. From techno- logical point of view, work with an interaction model completely touchless (no mouse paradigm was used) on small screen highlights all the limits of actual technologies. Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. From methodological outlook, this project underline the importance of feedback to the users first, during and after they interact with the system. The small size of the screen make this very complex and only a optimization of the UI has allowed to obtain a reli- able and usable system. 5.2 Large screen: Kinect Story Teller The case study is a multimedia application for the museum temporary exhibition “Man Ray”, held at the Art Museum of Lugano (Switzerland) in 2011 and presenting a vast selection of works by and about the photographer and painter Man Ray, one of the most influential artists of the twentieth century. The application was initially conceived as a multimedia narrative tool for desktop and tablet devices (http://www.manraylugano.ch/it/multimedia.html), to be used by visitors before, dur- ing and after the visit. The Gesture Touchless version was developed later, for being installed in the museum as a “memory” of the exhibition after its end. This application has been designed for attracting people’s attention when entering the museum, improv- ing the museum brand by offering a token of a past exhibition, complementing the physical cultural experience of the museum visit with a virtual cultural experience. The three applications (web, tablet and gesture based) share the set of content units – texts, audios, videos, panoramic images, floor maps - and logical information structure. The application is conceptually divided into two parts. Fig. 5. First part of interaction In the first part of the experience, users explore the map of the museum using their hands as pointers. This kind of paradigm is very familiar for the users and immediately understands how interact with the system. Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. Fig. 6. Second part of application When users select a particular room, start the second part of application: users view a 360° image of the room and when they move their shoulders all the room rotate and this generate surprise and fun. This project has allowed me to deal with other aspects of my research, in particular, how create an immersive experience for the users and how transfer contents from an existing application based on traditional paradigm of interaction. 5.3 Motion-based Touchless Games for Autistic Children Our understanding of the effectiveness of motion-based touchless games for autistic children is limited, because of the small amount of empirical studies and the limits of our current knowledge on autism. [17] offers two contributions. First, I provide a survey and a discussion of the existing literature. Second, we describe a field study that extends the current body of empirical evidence of the potential benefits of touchless motion- based gaming for autistic children. Our research involved five autistic children and one therapist in the experimentation of a set of Kinect games at a therapeutic center for a period of two and a half months. Using standardized therapeutic tests, observations during game sessions, and video analysis of over 20 hours of children’s activities, we evaluated the learning benefits in relationship to attentional skills and explored several factors in the emotional and behavioral sphere. While several authors assume that embodied touchless interaction will help improve skills of autistic children, the mechanism is not clear and whether touchless gaming is appropriate with these special users is a challenging research question. This paper con- tributes to a better understanding of this open issue. The findings of our study provide some empirical evidence that motion-based touchless games can promote attention Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. skills for autistic children with low-moderate cognitive deficit, low-medium sensory- motor dysfunction, and motor autonomy. In a relatively short time the participants to our study could learn how to use touchless gestures for play purposes, and could be- come autonomous players; as the gaming experience proceeded, stringer positive emo- tions were triggered and distress tended to decrease, moderating the negative effects that “breaks of routine” normally induce on autistic children. All these results have to be considered tentative. We don’t know the degree to which the measured benefits represent a persistent achievement and what we have measured in a specific setting can be translated to other contexts and moments of participants’ life. Our research design has some flaws, as five different stimuli were given in a series, without returning to a baseline measure. The causality of the improvements is hard to define, as we could not isolate all variables that may influence the learning process. We cannot conclude that the benefits we detected have to be ascribed to the motion-based touchless interaction paradigm, the contents of games and the visual design, or a com- bination of these and other factors. Even if no other therapeutic treatment was admin- istrated to our children during the study period, other activities that the children expe- rienced in these 2.5 months could have influenced our evaluation. Finally, our work has involved five children only - a small sample, but comparable to the sample size of most existing research addressing autistic children’s in relationship to technology, and quite a standard number in applied behavioral analysis. Considering the wide range of ASD impairments, more research is needed both to confirm our results for subjects having profile a similar to the subjects involved in our study, and to translate our findings to other types of autistic children. In spite of all the above limitations, the research reported in this paper sheds a light on how autistic children behave when engaged in motion-based touchless gaming. It is a first step in an exploratory process for identifying how to design motion based touch- less playful experiences for autistic children, and how to use them for therapy and ed- ucation. 6 Conclusions These case studies show the variety of applications that this research seeks to ad- dress. Although seemingly different, all these areas share the same technological and conceptual problems. The next steps of the research will improve the technology frame- work and try to conceptualize in a single methodological framework all accumulated experiences in the different case studies. 7 References 1. Van Aart C., Wielinga B., van Hage W. R. (2010). Mobile Cultural Heritage Guide: Loca- tion-Aware Semantic Search. Knowledge Engineering and Management by the Masses, Lec- ture Notes in Computer Science, Volume 6317/2010, 257-271. Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. 2. Davies N., Friday A., Newman P., Rutlidge S., Storz O. (2009). Using bluetooth device names to support interaction in smart environments. MobiSys 09. Proc. 7th Int. Conf. on Mobile Systems Applications and Services, ACM, 151-164. 3. Mitchell K., Race N., Suggitt M. (2006). iCapture: Facilitating Spontaneous User-Interac- tion with Pervasive Displays using Smart Devices. In: Workshop on Pervasive Mobile In- teraction Devices - Mobile Devices as Pervasive User Interfaces and Interaction Devices (PERMID) 4. Alisi T. M., D'Amico G., Ferracani A., Landucci L., Torpei N. (2010). Natural interaction for cultural heritage: the archaeological site of Shawbak. Proc. Int. conference on Multime- dia, ACM, 1449-1452. 5. Wobbrock J.O., Morris M. R., Wilson. A.D. (2009). User-defined gestures for surface com- puting. Proc. 27th Int. Conf. on Human factors in computing systems (CHI '09), ACM, 1083- 1092. 6. http://multi-touch-system.com/ 7. J.C. Lee. Hacking the Nintendo wii remote. (2008). Pervasive Computing, IEEE, 7(3):39 - 45. 8. Schlomer T., Poppinga B., Henze N., Boll S. Gesture recognition with a wii controller. Proc. 2nd Int. Conf. Tangible and Embedded Interaction (TEI '08), ACM, 11-14. 9. Mathieu Hopmann, Patrick Salamin, Nicolas Chauvin, Frédéric Vexo, and Daniel Thal- mann. 2011. Natural activation for gesture recognition systems. Proc. of annual Conf. ex- tended abstracts on Human factors in computing systems (CHI EA '11), ACM., 173-183. 10. Bannon L., Benford S., Bowers, J., & Heath C. (2005). Hybrid design creates innovative museum experiences. Comm. of the ACM, 48(3), 62-65. 11. Koleva B., Egglestone S. R., Schnädelbach H., Glover K., Greenhalgh C., Rodden T., & Dade-Robertson M. (2009). Supporting the creation of hybrid museum experiences. Proc. of the 27th Int. Conf. on Human factors in computing systems CHI 09, ACM Press, 1973. 12. Karen Holtzblatt, Jessamyn Burns Wendell, Shelley Wood. (2006). Rapid contextual design: a how-to guide to key techniques for user-centered design, Morgan Kaufman. 13. Villaroman, N., Rowe, D., Swan, B. 2011. Teaching natural user interaction using OpenNI and the Microsoft Kinect Sensor. Proc.SIGITE 2011, 227-232. ACM 14. Grandhi, S. A., Joue, G., Mittelberg, I. 2011. Understanding naturalness and intuitive- ness in gesture production: insights for touchless gestural interfaces. Proc. CHI 2011, 821- 824. ACM. 15. Nielsen M., Störring, M., Moeslund, T., Granum E. 2004. A procedure for developing intu- itive and ergonomic gesture interfaces for HCI. Gesture-Based Communication in HCI, 105- 106. Springer. 16. Malizia, A. and Bellucci, A. The artificiality of natural user interfaces. CACM 55, 3 (March 2012), 36-38. 17. L. Bartoli, C. Corradi, F. Garzotto and M.Valoriani, Exploring Motion-based Touchless Games for Autistic Children’s Learning. IDC 2013 Proc. of CHItaly 2013 Doctoral Consortium, Trento (Italy), September 16th 2013 (published at http://ceur-ws.org). Copyright © 2013 for the individual papers by the papers' authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.