The objective of the research is to build technological and methodological solutions to support the development of interactive applications that enable a fluid interaction with multimedia contents. The objective is pursued employing different paradigms of content presentation, interaction and control, supporting individual and social tasks, and through customization with respect to the characteristics of the intended users and situations of use. The case studies for understanding user requirements and for applying the research results, are envisioned in the domain of cultural heritage, tourism and healthcare.
The recent evolution of ICT allows for the creation of innovative User Experiences (UXs) that, at individual and collective level, expand our access to and interaction with digital information in a way that would have been unconceivable just few years ago.
In this arena, the research will address the design and development of novel
interactive applications placed at the intersection of three macro categories: mobile [
Applications in the mobile category are those delivered on hand-held devices, for
instance smartphones or tablets, which make contents and services available
anywhereanytime. In the second category there are applications involving multi-touch large
surfaces (as Microsoft Pixelsense[
This project explores problems and challenges arising from a new kind of
interaction: touchless interaction. This category, probably the most heterogeneous one,
comprises applications developed in such a way that user interaction does not require
physical contact with input/output devices, but is intangible, and exploits technologies such
as voice, gesture, or body movement recognition (e.g., Wii[
The ultimate objective is to build technological and methodological solutions to support the development of interactive applications that enable a fluid interaction with multimedia content, support individual and social tasks, and facilitate the customization with respect to the characteristics of the intended users and situations of use.
In order to include the research in tangible context of use, two areas will be addressed: Cultural heritage/tourism and Healthcare and disabilities support. 1.1
This domain involves a variety of user profiles, needs, situations of use, and a large
amount of information (think of museums, libraries, indoor or outdoor exhibitions,
monuments, archeological sites, …). This is a particularly rich and stimulating field
where the integration of different contents, devices and interaction paradigms,
combined with personalization and contextualization features, is particularly promising
with respect to the creation of more compelling and unique experiences for end users
[
Limited research in this arena has explored the potential of “motion-based” (or
“fullbody”) touchless interaction. This paradigm exploits sensing devices, which capture,
track and decipher body movements and gestures without users needing to wear
additional devices (e.g., data gloves, head mounted display, remote controllers, or body
markers). Several authors claim that motion-based touchless interaction has the
potential to be more ergonomic and “natural” than other forms of interaction [
The work must face a number of research challenges. First of all, there is a
Technological Challenge, since:
1. Different users perform gesture in different ways;
2. Context changes the meaning of users’ gesture; [
A second important issue to be addressed is the UX (User Experience) Design Challenge. Gestures cannot be viewed as standard and the design of gesture is fundamental in order to create a “natural” user experience. From this point of view, the critical requirements are related to personalization, contextualization, and natural interaction of applications. A UX is more engaging and effective if multimedia contents and interaction mechanisms are customized with respect to the actual user(s) profile and her needs, and when these elements are appropriate for the actual “context” (e.g., the user’s physical disease, the situation where the UX takes place, the characteristics of the
physical location, and the social circumstances - individual vs. multi-user). Mastering all these aspects increases the complexity of the UX design space.
It is therefore important to identify a conceptual modeling framework, which plays the same role as conceptual models in other areas, such as web applications. A modeling framework identifies the dimensions along which the multi-device/multi-paradigm personalized/ contextualized UXs can be designed providing designers and developers with a “language” to specify the characteristics of the applications under design, as well as guidelines to organize their work; in addition, it offers a taxonomy for the quality variables to be measured while evaluating the final UXs.
Technology UX
App. domain Modeling
Contextual Studies [
Definition of the user requirements (in terms of contents, interaction, personalization, and contextualization needs) for multi-device coordinated applications in the cultural heritage domain.
application execution.
recognition.
dleware’s capabilities and offers case studies for its technical tuning and testing.
User testing of such application(s), first in a controlled laboratory environment, and them in a real context (e.g., a real museum or therapeutic centers).
Based on the experience gained in the previous tasks, definition of a conceptual framework (comprising guidelines, modeling concepts and primitives) to support the design and development of the considered class of applications. 4
Currently the writer is developing a framework for identifying and interpreting the users’ gestures inserting also fuzzy rules in order to decrease the dependence on parameters and make the system more flexible. I intend to try to apply some algorithms of ANN to modify dynamically the weights of my network. There are two main components in the framework: the GestureRecognizer and the GestureManager (Figure 2).
The GestureRecognizer dynamically analyses a subset of all body skeleton data detected by the sensing device (considering only information related to torso and upwards joints) and progressively checks if such data correspond to a gesture included in the set of the possible ones. A 2D active zone, a start condition, a set of constraints, and an end condition model each gesture. For example, a user “horizontal swipe of the right hand” (Figure 3) movement is interpreted as “Horizontal Browsing” if: 1. the right hand is detected in the area identified by the horizontal line “from left shoulder to twice the distance among shoulders” and the vertical line from “from left shoulder to the center of mass (belly) line” (active zone);
the right hand is moving horizontally (start condition); the motion continues with minimum speed V (constraint); the traversed distance is > D (end condition).
The GestureRecognizer implements small modules to recognize some generic characteristics of movement. Composing these modules as a network, it is possible to model and recognize many gestures reusing the same code.
The GestureRecognizer is fully parametric so that the designers can easily change the setting of gesture specification parameters to adapt them to the anthropometric characteristics of a specific user.
Once a gesture has been recognized, its data are transmitted to the GestureManager for interpretation and execution. The semantics of a gesture (i.e., its effect on the application state) depend on the current execution context. The GestureManager checks for gesture intentionality and it executes the corresponding application state. 5
This section includes a fast overview of a selection the projects developed in the context of this research in order to provide a better understanding of the different aspects of the research. 5.1
Touchless gestural interaction in-the-small has received so far only a marginal
attention in industry and research. Still, this paradigm has a potential in a number of
interesting domains. In addition, it is challenging from a research point of view. Building
any touchless gesture-based application has an intrinsic complexity related to both
achieving accurate and meaningful gesture recognition and identifying natural, intuitive
and meaningful gesture vocabularies appropriate for the tasks under consideration [
When dealing with touchless gestural interaction in the small, this complexity is exacerbated by the fact that gestures and their detection must be designed in relation to visual interfaces that face a number of constraints:
i) size; iii) number of interface elements and contents that can be displayed, and their dimension; digital space available to provide feedbacks to user’s interaction.
MOTIC (MOtion-based Touchless Interactive Cooking system) is a case study in which I have applied touchless gestural interaction in the small to the domain of household appliances. MOTIC comprises a conventional oven that is integrated with a small display and a Kinect motion sensing device (Figure 4) to enable users control cooking behavior thought body motion and touchless gestures. The design of any system based on gestural interaction involves two main dimensions. One concerns the definition of the gesture language, which specifies the shape (trajectory) and dynamics of body movements that express the user’s intentions, and communicate to a system while performing the tasks identified during requirement elicitation. The second design dimension is related to the specification of the visual interface on display, presenting information and functions that the user can operate on and feedbacks on interaction. Finally, in applications requiring a smooth integration of the gestural system with the physical environment, a third design dimension needs to be addressed, concerning the actual positioning of hardware devices (sensing device and display). This dimension, in the case of MOTIC, is related the actual positioning of the Kinect within the home kitchen furniture (being the display position fixed).
To address the three design dimensions, we have followed an iterative user-centered design process during which we have repeatedly defined, implemented, and tested gestures, visual interfaces, and Kinect positions, using progressively more sophisticated artifacts (from paper based mockups to functional prototypes) and involving users along the entire process.
In my research this has been very important to understand two aspect. From technological point of view, work with an interaction model completely touchless (no mouse paradigm was used) on small screen highlights all the limits of actual technologies.
From methodological outlook, this project underline the importance of feedback to the users first, during and after they interact with the system. The small size of the screen make this very complex and only a optimization of the UI has allowed to obtain a reliable and usable system. 5.2
The case study is a multimedia application for the museum temporary exhibition “Man Ray”, held at the Art Museum of Lugano (Switzerland) in 2011 and presenting a vast selection of works by and about the photographer and painter Man Ray, one of the most influential artists of the twentieth century. The application was initially conceived as a multimedia narrative tool for desktop and tablet devices (http://www.manraylugano.ch/it/multimedia.html), to be used by visitors before, during and after the visit. The Gesture Touchless version was developed later, for being installed in the museum as a “memory” of the exhibition after its end. This application has been designed for attracting people’s attention when entering the museum, improving the museum brand by offering a token of a past exhibition, complementing the physical cultural experience of the museum visit with a virtual cultural experience. The three applications (web, tablet and gesture based) share the set of content units – texts, audios, videos, panoramic images, floor maps - and logical information structure. The application is conceptually divided into two parts. In the first part of the experience, users explore the map of the museum using their hands as pointers. This kind of paradigm is very familiar for the users and immediately understands how interact with the system. When users select a particular room, start the second part of application: users view a 360° image of the room and when they move their shoulders all the room rotate and this generate surprise and fun.
This project has allowed me to deal with other aspects of my research, in particular, how create an immersive experience for the users and how transfer contents from an existing application based on traditional paradigm of interaction. 5.3
Our understanding of the effectiveness of motion-based touchless games for autistic
children is limited, because of the small amount of empirical studies and the limits of
our current knowledge on autism. [
While several authors assume that embodied touchless interaction will help improve skills of autistic children, the mechanism is not clear and whether touchless gaming is appropriate with these special users is a challenging research question. This paper contributes to a better understanding of this open issue. The findings of our study provide some empirical evidence that motion-based touchless games can promote attention
skills for autistic children with low-moderate cognitive deficit, low-medium sensorymotor dysfunction, and motor autonomy. In a relatively short time the participants to our study could learn how to use touchless gestures for play purposes, and could become autonomous players; as the gaming experience proceeded, stringer positive emotions were triggered and distress tended to decrease, moderating the negative effects that “breaks of routine” normally induce on autistic children.
All these results have to be considered tentative. We don’t know the degree to which the measured benefits represent a persistent achievement and what we have measured in a specific setting can be translated to other contexts and moments of participants’ life. Our research design has some flaws, as five different stimuli were given in a series, without returning to a baseline measure. The causality of the improvements is hard to define, as we could not isolate all variables that may influence the learning process. We cannot conclude that the benefits we detected have to be ascribed to the motion-based touchless interaction paradigm, the contents of games and the visual design, or a combination of these and other factors. Even if no other therapeutic treatment was administrated to our children during the study period, other activities that the children experienced in these 2.5 months could have influenced our evaluation. Finally, our work has involved five children only - a small sample, but comparable to the sample size of most existing research addressing autistic children’s in relationship to technology, and quite a standard number in applied behavioral analysis. Considering the wide range of ASD impairments, more research is needed both to confirm our results for subjects having profile a similar to the subjects involved in our study, and to translate our findings to other types of autistic children.
In spite of all the above limitations, the research reported in this paper sheds a light on how autistic children behave when engaged in motion-based touchless gaming. It is a first step in an exploratory process for identifying how to design motion based touchless playful experiences for autistic children, and how to use them for therapy and education. 6
These case studies show the variety of applications that this research seeks to address. Although seemingly different, all these areas share the same technological and conceptual problems. The next steps of the research will improve the technology framework and try to conceptualize in a single methodological framework all accumulated experiences in the different case studies. 7