Dozens of novel natural interaction techniques are proposed every year to enrich interactive eco-systems with multitouch gestures, motion gestures, full body in motion, etc. We present a novel investigation of the community's applied documentation practices for Natural User Interfaces (NUI). Our investigation includes analyzing a survey targeted at NUI designers and a large sample of recently published multitouch and motion-based interaction papers. To the best of our knowledge, this paper is the first to offer a close investigation of this kind. The results reveal that good NUI documentation practices are rare and largely compromised. Thus, we argue that engineering interactive systems for large-scale dynamic runtime deployment of existing interaction techniques is greatly challenged.
INTRODUCTION
Calls arise to explore new potential in designing for the whole
body in motion as part of the NUI paradigm [
Copyright c 2015 for the individual papers by the papers’ authors. Copying permitted only for private and academic purposes. This volume is published and copyrighted by its editors. Herein, we focus on a subset of Kinetic-based interactions, namely multitouch- and motion-based interactions. In the last decade, touch and motion enabled technologies found their way commercially and became widely accessible to the end user, in various application domains such as gaming (e.g., motion-controlled active play by Microsoft Kinect1 or the Wii system2), data browsing, navigation scenarios (e.g., tilting for scrolling photos as in iOS3 and Android4 devices) and many more.
Despite the immense progress and success in different
application domains, interactive environments will pose
additional significant challenges to the design, engineering and
deployment of NUI technologies. Considering user
heterogeneity, e.g. due to aging and demographic change
(”comeas-you-are” paradigm), user mobility to unknown
environmental settings at design time (interaction context) and
spontaneous construction of interactive environments in-situ at
runtime, the isolated design of natural interface devices will
not be sufficient any more, regardless of the quality and
naturalness of the proposed interaction scheme per se. In
their work, Altakrouri and Schrader [
In this paper, the term documentation is used to capture the
way an interaction technique is defined and described by the
interaction designer (i.e. developer). Principally,
documentation refers to any written material, visual clues, animated
clues, formal description models and languages, etc, used to
describe or disseminate the developed interaction. The
literature covers various approaches to describe touch-based
interactions. An extensive review on those approaches is out of
the scope of this paper. In their work about formal
descriptions for multitouch interactions, Hamon et al. [
Better understanding of the currently applied documentation practices does not only reveal the current dissemination strategies but also triggers possible needs for new tools, guidelines, and systems that improve those practices and ultimately bridge the gap between the design of single interaction techniques and the development of interactive systems. In this paper we will substantiate the following main contributions and findings:
We present a number of observations regarding the NUI designers’ most commonly applied documentation choices, most importantly, documentation frequency and media type of choice.
We unveil that NUI documentation is largely underestimated and compromised by NUI designers due to the lack of adequate documentation tools, absence of documentation standards, and irregularity of documentation habits. METHODOLOGY Our study included two investigation areas: (1) analyzing a tailored survey targeted at NUI designers and (2) coding and analyzing a large sample of recently published multitouch and motion-based interaction papers. In this section, we first outline our approach before we present the results in the following section.
Survey on NUI documentation The first step in our review was to capture a snapshot on the current most employed practices for NUI documentation by carrying out an online survey. The survey aimed to partially characterize a number of designers’ documentation practices, including: (1) The adoption level and frequency of documentation practices and standards in design and development of NUI; (2) The designers’ satisfaction with their practiced NUI documentation habits; (3) The needs for new documentation tools and methods; (4) The commonly used documentation methods, tools, and media types; and (5) The perceived importance of documentation for sharing, acceptance, user experience, and correctness.
The survey was targeted at both NUI designers (i.e. NUI developers) and users, it was split into two sections accordingly. In this paper, we only focus and report about the designer section, which contained a total sum of 11 different multiple choice and likert scale questions. The survey was bound to a maximum completion time of 3 minutes to maximize the number of voluntary participations. The survey included an introductory section where the notion of NUI, specially for multitouch- and motion-based interfaces, as well as the purpose of the survey were introduced.
The survey was distributed online through specialized HCI mailing lists (including BCS-HCI run by the British Computer Society Human-Computer Interaction Group5), ubiquitous computing mailing lists (including Ukubinet-announce run by the Imperial College London6 and announcements@ubicomp.org7), Lu¨beck university mailing lists, and social networks (i.e. Facebook, Twitter, and ResearchGate). The survey was open for participation for about 3 weeks. Analyzing the interaction publications landscape The second step in our review intended to capture a closer look at the published work in the area of interaction techniques. In order to find out how the community expresses, documents, and shares interaction techniques, we have decided to base our investigation on a collection of the most recent ACM published work under the ACM classification (H.5.2 Information Interfaces and Presentation: User Interfaces - Input devices and strategies) for the years 2012 and 5https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=bcs-hci, latest access on 25.03.2015. 6https://mailman.ic.ac.uk/mailman/listinfo/ukubinet-announce, latest access on 25.03.2015. 7http://mail.ubicomp.org/mailman/listinfo/announcements ubicomp.org, latest access on 25.03.2015. 2013 (until 22.08.2013). Out of 518 total papers in this category, we manually coded and analyzed a total sum of 93 papers that matched one of two categories: (1) papers presenting novel interaction techniques; (2) papers applying or analyzing existing interaction techniques in various scenarios. Our filtering criteria excluded all none touch or none motion gesture papers (as considered out of the focus of this investigation), video papers (as those papers don’t have enough space to cover the interaction technique and only convey very limited aspects of the work), and duplicated paper entries (if the same work was presented in multiple venues but with different contribution size, e.g., work-in-progress papers, short papers, full papers). In the case of duplication, the latest and longest contribution was considered. Our aim was not to conduct a complete and detailed review of all published papers. Instead, we aimed at providing a snapshot at the most recent published work as a living example of the current practiced documentation habits.
Our analysis and classification are based on the published paper and any corresponding material directly mentioned, linked, or attached with the published work (e.g., many published papers have also videos attached within the ACM library, or links to external resources). Other materials out of the aforementioned criteria were considered hidden and were not included in the study, such as in application help menus or offline accessible manuals.
The papers were coded based on four main aspects: Type - gesture types discussed in the paper including multitouch and motion gestures; Still - used still media types to document and describe the gesture including text, images, and sketches; Animated - used animated media types to document and describe the gesture including videos, animations, personal walkthrough, and onscreen walkthrough; and finally Authoring - reported or used authoring and documentation tools and formal languages. Our main goal of this analysis was to highlight general practices and habits rather than focusing on a particular paper title or the authors. Hence, we reference the reviewed papers by the unique identification key (ACM ID) instead of the papers’ full title or author names. RESULTS AND OBSERVATIONS In this section, we present the results for each of our investigation areas. We have supported the data with a number of general observations to enhance the readability of the results. The observations are numbered and marked with an abbreviation to the corresponding section (D: Designer survey section and P: Papers analysis section).
Survey A total of 332 anonymous individual responses were recorded, split into 267 NUI users (80% of the total respondents) and 65 NUI designers (20% of the total respondents). The designer respondents are split to 11 expert designers, 14 professional designers, 28 competent designers, 10 advanced beginners, and finally 2 novice designers. This categorization is based on an explicit survey question about expertise selfassessment.
We have applied Kruskal Wallis test to identify any statistically significant differences among expertise groups. In most cases, no statistical differences amongst groups were found unless explicitly mentioned in the text.
Observation - D1: Small majority of NUI designers are satisfied with their current documentation practices: 57% of the designers responded positively to a question on the satisfaction with their current documentation habits.
Observation - D2: Only a small minority of NUI designers practice NUI documentation continuously: Figure 1 shows how often the designer respondents document designed NUI, independent of form or documentation type. The figure reveals that the majority of the respondents practice documentation either sometimes (42%) or frequently (38%). Merely small minority of designers (14%) practice documentation regularly. Statistically significant difference was identified among expertise (H(4) = 13.466, p = 0.009) with a mean rank of 43.93 for proficient, 33.75 for competent, 32.91 for expert, 18.70 for advanced beginner, and 18.0 for novice designers. Higher mean ranks indicate a more frequent documentation practice. Observation - D3: The vast majority of NUI designers never or rarely apply documentation standards: One interesting aspect in this survey is to highlight the designers’ habits to apply standard documentation approaches, as shown in Figure 1. The survey unveils that about half of the respondents never apply any documentation standards and merely a third did on rare occasions. Small number of respondents apply documentation standards either sometimes (14%) or frequently (3%). Observation - D4: The majority of NUI designers indicated a lack of NUI documentation tools and methods: The respondents answered positively (66%) when asked whether there is a lack of NUI documentation methods and tools available for them to use.
Observation - D5: NUI are mostly documented using text, pictures, sketches, and videos respectively: Another goal of the survey was to identify the dominant media types used by designers to document interaction techniques. Figure 2 illustrates the distribution of used NUI documentation media types by designers. Text is the most used medium to describe and document NUI. Still visual documentation records (i.e. pictures and sketches) follow next. Moreover, animated visual records come fourth. Additionally, audio and formal languages come last with very low percentages. Observation - D6: NUI are rarely documented using formalized languages: Figure 2 also shows clearly designers don’t follow formalizations as a documentation media type. Observation - D7: The most ranked importance of NUI documentation is acknowledged for sharing NUI, followed by user experience: Figure 3 illustrates the designers’ perceived importance of NUI documentation for sharing, experience, acceptance, and correctness. The vast majority of responders scored documentation as a very important (45%) or an important (37%) factor for a successful sharing of NUI. Regarding user experience, the majority of respondents scored the documentation as an important (48%) or a very important (11%) factor respectively. Moreover, designers scored NUI documentation for user acceptance as very important (14%), important (40%), moderate (25%), and of little importance (18%). Merely 3% negatively scored documentation as unimportant for the user acceptance. Finally, the majority of respondents scored documentation as either an important (40%) or very important (26%) factor for the correctness of NUI execution. Approximately one third of the respondents scored documentation as moderate or of little importance for correctness.
Scientific publications Figure 4 illustrates the complete classification of the analyzed papers based on the previously presented methodology. Papers that satisfy the conditions are distinguished with a coding mark as shown in the figure. The analyzed papers were motion (51%) and touch based (68%) interaction papers (note that a paper may fall into more than one category).
Observation - P1: NUI in publications are mostly documented using text, sketches, and pictures respectively: As expected, figure 4 shows that text descriptions as a medium for documenting interaction techniques is used in all of the papers that we have reviewed. Sketches (59%) come second with a very close match with the designer survey in Figure 2. Pictures (53%) come third, slightly lower than in the designer survey.
Moreover, personal walkthrough is reported by 16% (the developer introduces the interaction technique to other developers or users by demonstration). Videos are reported by 11%. This percentage matches the survey’s results (Figure 2). In research papers, mentioning and linking to video content is usually neither required nor critical for the acceptance of the research paper. Hence, videos related material to the technique are often hidden. The use of animations is reported only once. This matches to a large extend the designer survey results in Figure 2. On the other hand, other media types such as onscreen walkthrough are hardly used.
Observation - P2: NUI in publications are never documented using formalized languages or interaction authoring tools: To our expectations, none of the papers reported or used languages (including notations and formalisms) or interaction authoring tools (including gesture authoring tools). Finally, we found no statistical difference between the two main aforementioned analyzed groups of papers.
DISCUSSION In this section, we present a number of interesting aspects regarding NUI documentation practices and possible impact on designing and engineering interactive systems.
Documentation habits: Ignorance or underestimation? Our results show that the majority of the designer respondents are satisfied with their current NUI documentation practices (D1). Nonetheless, this satisfaction is not necessarily reflected on the quality and extend of applied documentation practices (D2, D3). Those observations unveil that NUI documentation is generally an underestimated or ignored problem by interaction designers and developers.
Clearly, the NUI paradigm vastly grows in terms of the
number of interaction proposed, the diversity of interaction types,
involved body parts, involved actions, etc. [
Soon the lack of adequate interaction documentation and dissemination will lead to challenge the design and engineering of interactive systems. Documentation can be used to extract information about the type of movements involved in the interaction, involved body parts, adequate interaction execution, etc. The absence of such information will necessarily lead to burden the deployment of interaction techniques in automated interactive systems, especially processes such as context acquisition, reasoning, interaction filtering, etc. are greatly hindered. The absence of documentation inevitably challenges analyzing the proposed interaction techniques. For instance, requirement analysis can be greatly compromised, analyzing physical requirements for NUI users is not possible, or correctly reproduce or extend a particular interaction techniques becomes an extremely challenging task.
Documentation types and methods The results show that various media types are used by designers for documentation. From one hand, designers reported that the most used media types for documenting NUI are text, pictures, sketches, and videos respectively (D5). The academic paper investigation on the other hand shows text, sketches, and pictures respectively used in the analyzed papers (P1). Despite the aforementioned difference, it is clearly visible that designers rarely use formalized languages or approaches as a documentation medium (D3, D6, P2). Formalized description of NUI are an important mean to document various aspects on the interaction to insure integrity and correctness of execution, and the possibility of replicability. Designers are not in favor of formal languages and audio. This can be due to the complexity of language learning, and the complexity of describing movements respectively. Whilst formalized languages can be hard to learn and apply, they are often used in different fields for documenting movements. Formalized languages have a clear benefit to preserve and transfer the technique to other designers without endangering the originality and vital aspects of the technique. Using the currently applied media such as text, pictures, sketches, and videos may lead easily to losing parts of the movements, overly complicated descriptions, losing timing information, etc.
In fact, according to Navarre et al. [
The notion of movement is of particular importance for
Kinetic Interactions, as it resides at the core of this type of
interactions. Movement documentation is a very relevant and
generally a very unresting problem for many fields such as dance
choreography, movement rehabilitation, motion recognition
and analysis, and human movement simulation.
According to Kahol et al. [
Current NUI documentation practices, as discussed in this paper, are greatly challenged by such a system. The current documentation practices and strategies are not adequate and fail to meet the challenge of dynamically created documentation for interaction ensembles. Interactions are currently ego-centric and designed in isolations, so is the documentation. Such isolation implies a complete absence of information about the interaction’s behavior as part of an ensemble in a dynamically changing eco-system.
FUTURE WORK As part of our research roadmap, we will continue to explore this field by (1) extending our investigation to study the differences and similarities between NUI documentation in academic and commercial settings (as in motion-based and touch-based application market initiatives); (2) extending our analysis to include NUI users and their learning habits; and (3) extending our ongoing work on a dedicated tool for documenting NUI CONCLUSION We have presented an investigation on the applied practices and habits to document and share developed interaction techniques. The analysis included: (1) an online exploratory survey on documenting Natural User Interfaces (NUI) answered by 64 designer; and, (2) coding and analyzing a sample of 93 recently ACM published multitouch and motion-based interaction papers. Our study reveals that good documentation practices are rare and largely compromised. Our survey reveals that there is a lack of documentation tools, methods, and formal languages; designers almost never follow or apply any documentation standards; and designers never use available interaction authoring tools. Hence, the creation of a collective long lasting interaction heritage remains unachievable. Moreover, the gap between developing and rightly disseminating interaction techniques increases. Thus, engineering interactive systems for large-scale dynamic runtime deployment of existing and future interaction techniques is greatly challenged.