Challenging Documentation Practices for Interactions in Natural User Interfaces Bashar Altakrouri Andreas Schrader Ambient Computing Group Ambient Computing Group Institute of Telematics Institute of Telematics University ofLübeck University of Lübeck Lübeck, Germany Lübeck, Germany altakrouri@itm.uni-luebeck.de schrader@itm.uni-luebeck.de ABSTRACT Herein, we focus on a subset of Kinetic-based interactions, Dozens of novel natural interaction techniques are proposed namely multitouch- and motion-based interactions. In the last every year to enrich interactive eco-systems with multitouch decade, touch and motion enabled technologies found their gestures, motion gestures, full body in motion, etc. We way commercially and became widely accessible to the end present a novel investigation of the community’s applied doc- user, in various application domains such as gaming (e.g., umentation practices for Natural User Interfaces (NUI). Our motion-controlled active play by Microsoft Kinect1 or the Wii investigation includes analyzing a survey targeted at NUI de- system2 ), data browsing, navigation scenarios (e.g., tilting for signers and a large sample of recently published multitouch scrolling photos as in iOS3 and Android4 devices) and many and motion-based interaction papers. To the best of our more. knowledge, this paper is the first to offer a close investiga- Despite the immense progress and success in different ap- tion of this kind. The results reveal that good NUI documen- plication domains, interactive environments will pose addi- tation practices are rare and largely compromised. Thus, we tional significant challenges to the design, engineering and argue that engineering interactive systems for large-scale dy- deployment of NUI technologies. Considering user hetero- namic runtime deployment of existing interaction techniques geneity, e.g. due to aging and demographic change (”come- is greatly challenged. as-you-are” paradigm), user mobility to unknown environ- mental settings at design time (interaction context) and spon- Author Keywords taneous construction of interactive environments in-situ at Natural User Interfaces (NUI); Gesture Interfaces; Motion runtime, the isolated design of natural interface devices will Interfaces; HCI modeling; HCI documentation; HCI sharing. not be sufficient any more, regardless of the quality and naturalness of the proposed interaction scheme per se. In ACM Classification Keywords their work, Altakrouri and Schrader [2] proposed a shift to- H.5.m. Information Interfaces and Presentation (e.g., HCI): wards completely dynamic on-the-fly ensembles of interac- Miscellaneous; H.5.2 Information interfaces and presentation tion techniques at runtime. The Interaction Ensembles ap- (e.g., HCI): User Interfaces proach is defined as ”Multiple interaction modalities (i.e., in- teraction plugins) from different devices are tailored at run- INTRODUCTION time to adapt the available interaction resources and possibil- Calls arise to explore new potential in designing for the whole ities to the user’s physical abilities, needs, and context” [2]. body in motion as part of the NUI paradigm [3][4], to fa- A shift of this kind imposes new dissemination, deployment, cilitate users’ interactions with real-world pervasive ecosys- and adaptation requirements for engineering interaction tech- tems (ambient spaces). In the literature, different definitions niques and interactive systems for NUI. Precisely for those of NUI [6] were elaborated, which mostly refer to the user’s reasons, better understanding and analysis of the practiced natural abilities, practices, and activities to control interactive documentation habits of interaction techniques for NUI plays systems. Devised from Wachs et al. [10], interactions with a major role to bridge the possible gaps between designing NUI can be shortly defined as voice-based and kinetic-based interaction techniques and engineering interactive systems. interactions. Kinetic-based interactions are mostly caused and characterized by motion and movement activities, rang- In this paper, we present a novel investigation of the com- ing from pointing, clicking, grasping, walking, etc. [2] munity’s applied documentation practices for interactions in NUI. We believe that an investigation of this kind is essential to understand some of the challenges for engineering inter- active systems in ambient spaces and setting proper interac- 1 http://www.microsoft.com/en-us/kinectforwindows/, latest access Workshop on Large-scale and model-based Interactive Systems: Approaches and on 25.03.2015. 2 Challenges, June 23 2015, Duisburg, Germany. http://www.nintendo.com/wii, latest access on 25.03.2015. Copyright c 2015 for the individual papers by the papers’ authors. Copying 3 http://www.apple.com/ios/, latest access on 25.03.2015. permitted only for private and academic purposes. This volume is published and 4 copyrighted by its editors. http://www.android.com/, latest access on 25.03.2015. tion dissemination guidelines, where interactions are becom- METHODOLOGY ing increasingly dynamic, adaptive and multi-modal. Our study included two investigation areas: (1) analyzing a tailored survey targeted at NUI designers and (2) coding and Our novel investigation is concluded by analyzing a survey analyzing a large sample of recently published multitouch and targeted at NUI designers and a large sample of recently pub- motion-based interaction papers. In this section, we first out- lished multitouch and motion-based interaction papers. Al- line our approach before we present the results in the follow- though limited in scale, we believe that this investigation ing section. opens the door for important open research issues for the CHI and EICS community around this problem domain. In this paper, the term documentation is used to capture the Survey on NUI documentation way an interaction technique is defined and described by the The first step in our review was to capture a snapshot on the interaction designer (i.e. developer). Principally, documen- current most employed practices for NUI documentation by tation refers to any written material, visual clues, animated carrying out an online survey. The survey aimed to partially clues, formal description models and languages, etc, used to characterize a number of designers’ documentation practices, describe or disseminate the developed interaction. The liter- including: (1) The adoption level and frequency of documen- ature covers various approaches to describe touch-based in- tation practices and standards in design and development of teractions. An extensive review on those approaches is out of NUI; (2) The designers’ satisfaction with their practiced NUI the scope of this paper. In their work about formal descrip- documentation habits; (3) The needs for new documentation tions for multitouch interactions, Hamon et al. [5] analyzed tools and methods; (4) The commonly used documentation the expressiveness of various user interface description lan- methods, tools, and media types; and (5) The perceived im- guage (an extension to [8]) and suggested the ICO formalism portance of documentation for sharing, acceptance, user ex- for modeling multitouch interactions. Principally, modeling perience, and correctness. includes data description, state representation, event repre- sentation, timing, concurrent behavior, dynamic instantiation, The survey was targeted at both NUI designers (i.e. NUI de- etc. Recently, Altakrouri et al. [1] targeted their effort to velopers) and users, it was split into two sections accordingly. describe the movement aspects of motion-based gestures and In this paper, we only focus and report about the designer the physical context (i.e., abilities and disabilities) of the user. section, which contained a total sum of 11 different multiple choice and likert scale questions. The survey was bound to Documenting interaction technique is relevant for the correct a maximum completion time of 3 minutes to maximize the execution of interactions by end users, the preservation of number of voluntary participations. The survey included an technique by designers, the accumulation of knowledge for introductory section where the notion of NUI, specially for the community, and the engineering of interactive systems. multitouch- and motion-based interfaces, as well as the pur- We argue that documenting interactions should be treated as pose of the survey were introduced. an important resource of context information about the in- teraction technique, which can be also utilized by interactive The survey was distributed online through specialized HCI systems for various reasons. For instance, filtering relevant mailing lists (including BCS-HCI run by the British Com- interaction techniques at runtime in response to the user’s puter Society Human-Computer Interaction Group5 ), ubiqui- physical context (e.g., disabilities) as in the Interaction En- tous computing mailing lists (including Ukubinet-announce semble approach mentioned above. run by the Imperial College London6 and announce- ments@ubicomp.org7 ), Lübeck university mailing lists, and Better understanding of the currently applied documenta- social networks (i.e. Facebook, Twitter, and ResearchGate). tion practices does not only reveal the current dissemina- The survey was open for participation for about 3 weeks. tion strategies but also triggers possible needs for new tools, guidelines, and systems that improve those practices and ulti- mately bridge the gap between the design of single interaction Analyzing the interaction publications landscape techniques and the development of interactive systems. The second step in our review intended to capture a closer In this paper we will substantiate the following main contri- look at the published work in the area of interaction tech- butions and findings: niques. In order to find out how the community expresses, documents, and shares interaction techniques, we have de- • We present a number of observations regarding the NUI de- cided to base our investigation on a collection of the most signers’ most commonly applied documentation choices, recent ACM published work under the ACM classification most importantly, documentation frequency and media (H.5.2 Information Interfaces and Presentation: User Inter- type of choice. faces - Input devices and strategies) for the years 2012 and • We unveil that NUI documentation is largely underesti- mated and compromised by NUI designers due to the lack 5 https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=bcs-hci, latest of adequate documentation tools, absence of documenta- access on 25.03.2015. 6 tion standards, and irregularity of documentation habits. https://mailman.ic.ac.uk/mailman/listinfo/ukubinet-announce, lat- est access on 25.03.2015. 7 http://mail.ubicomp.org/mailman/listinfo/announcements ubicomp.org, latest access on 25.03.2015. 2013 (until 22.08.2013). Out of 518 total papers in this cate- We have applied Kruskal Wallis test to identify any statisti- gory, we manually coded and analyzed a total sum of 93 pa- cally significant differences among expertise groups. In most pers that matched one of two categories: (1) papers present- cases, no statistical differences amongst groups were found ing novel interaction techniques; (2) papers applying or an- unless explicitly mentioned in the text. alyzing existing interaction techniques in various scenarios. Our filtering criteria excluded all none touch or none motion Observation - D1: Small majority of NUI designers are sat- gesture papers (as considered out of the focus of this inves- isfied with their current documentation practices: 57% of the tigation), video papers (as those papers don’t have enough designers responded positively to a question on the satisfac- space to cover the interaction technique and only convey very tion with their current documentation habits. limited aspects of the work), and duplicated paper entries (if Observation - D2: Only a small minority of NUI designers the same work was presented in multiple venues but with dif- practice NUI documentation continuously: Figure 1 shows ferent contribution size, e.g., work-in-progress papers, short how often the designer respondents document designed NUI, papers, full papers). In the case of duplication, the latest and independent of form or documentation type. The figure re- longest contribution was considered. Our aim was not to con- veals that the majority of the respondents practice documen- duct a complete and detailed review of all published papers. tation either sometimes (42%) or frequently (38%). Merely Instead, we aimed at providing a snapshot at the most recent small minority of designers (14%) practice documentation published work as a living example of the current practiced regularly. Statistically significant difference was identified documentation habits. among expertise (H(4) = 13.466, p = 0.009) with a mean rank of 43.93 for proficient, 33.75 for competent, 32.91 for expert, Our analysis and classification are based on the published paper and any corresponding material directly mentioned, 18.70 for advanced beginner, and 18.0 for novice designers. linked, or attached with the published work (e.g., many pub- Higher mean ranks indicate a more frequent documentation lished papers have also videos attached within the ACM li- practice. brary, or links to external resources). Other materials out of the aforementioned criteria were considered hidden and were not included in the study, such as in application help menus or offline accessible manuals. The papers were coded based on four main aspects: Type - gesture types discussed in the paper including multitouch and motion gestures; Still - used still media types to doc- ument and describe the gesture including text, images, and sketches; Animated - used animated media types to document and describe the gesture including videos, animations, per- sonal walkthrough, and onscreen walkthrough; and finally Authoring - reported or used authoring and documentation tools and formal languages. Our main goal of this analysis was to highlight general practices and habits rather than fo- Figure 1. Practicing NUI Documentation and Complying with Stan- cusing on a particular paper title or the authors. Hence, we dards reference the reviewed papers by the unique identification key (ACM ID) instead of the papers’ full title or author names. Observation - D3: The vast majority of NUI designers never or rarely apply documentation standards: One interesting as- RESULTS AND OBSERVATIONS pect in this survey is to highlight the designers’ habits to ap- In this section, we present the results for each of our inves- ply standard documentation approaches, as shown in Figure tigation areas. We have supported the data with a number of 1. The survey unveils that about half of the respondents never general observations to enhance the readability of the results. apply any documentation standards and merely a third did on The observations are numbered and marked with an abbrevia- rare occasions. Small number of respondents apply documen- tion to the corresponding section (D: Designer survey section tation standards either sometimes (14%) or frequently (3%). and P: Papers analysis section). Observation - D4: The majority of NUI designers indicated a lack of NUI documentation tools and methods: The respon- Survey dents answered positively (66%) when asked whether there is A total of 332 anonymous individual responses were a lack of NUI documentation methods and tools available for recorded, split into 267 NUI users (80% of the total respon- them to use. dents) and 65 NUI designers (20% of the total respondents). Observation - D5: NUI are mostly documented using text, The designer respondents are split to 11 expert designers, 14 pictures, sketches, and videos respectively: Another goal of professional designers, 28 competent designers, 10 advanced the survey was to identify the dominant media types used by beginners, and finally 2 novice designers. This categorization designers to document interaction techniques. Figure 2 il- is based on an explicit survey question about expertise self- lustrates the distribution of used NUI documentation media assessment. types by designers. Text is the most used medium to describe and document NUI. Still visual documentation records (i.e. ing mark as shown in the figure. The analyzed papers were pictures and sketches) follow next. Moreover, animated vi- motion (51%) and touch based (68%) interaction papers (note sual records come fourth. Additionally, audio and formal lan- that a paper may fall into more than one category). guages come last with very low percentages. Observation - P1: NUI in publications are mostly docu- mented using text, sketches, and pictures respectively: As ex- pected, figure 4 shows that text descriptions as a medium for documenting interaction techniques is used in all of the pa- pers that we have reviewed. Sketches (59%) come second with a very close match with the designer survey in Figure 2. Pictures (53%) come third, slightly lower than in the designer survey. Moreover, personal walkthrough is reported by 16% (the de- veloper introduces the interaction technique to other develop- ers or users by demonstration). Videos are reported by 11%. This percentage matches the survey’s results (Figure 2). In research papers, mentioning and linking to video content is Figure 2. Medium for Documentation usually neither required nor critical for the acceptance of the research paper. Hence, videos related material to the tech- Observation - D6: NUI are rarely documented using formal- nique are often hidden. The use of animations is reported ized languages: Figure 2 also shows clearly designers don’t only once. This matches to a large extend the designer survey follow formalizations as a documentation media type. results in Figure 2. On the other hand, other media types such Observation - D7: The most ranked importance of NUI docu- as onscreen walkthrough are hardly used. mentation is acknowledged for sharing NUI, followed by user Observation - P2: NUI in publications are never documented experience: Figure 3 illustrates the designers’ perceived im- using formalized languages or interaction authoring tools: portance of NUI documentation for sharing, experience, ac- To our expectations, none of the papers reported or used ceptance, and correctness. The vast majority of responders languages (including notations and formalisms) or interac- scored documentation as a very important (45%) or an im- tion authoring tools (including gesture authoring tools). Fi- portant (37%) factor for a successful sharing of NUI. Re- nally, we found no statistical difference between the two main garding user experience, the majority of respondents scored aforementioned analyzed groups of papers. the documentation as an important (48%) or a very impor- tant (11%) factor respectively. Moreover, designers scored DISCUSSION NUI documentation for user acceptance as very important In this section, we present a number of interesting aspects (14%), important (40%), moderate (25%), and of little im- regarding NUI documentation practices and possible impact portance (18%). Merely 3% negatively scored documentation on designing and engineering interactive systems. as unimportant for the user acceptance. Finally, the majority of respondents scored documentation as either an important Documentation habits: Ignorance or underestimation? (40%) or very important (26%) factor for the correctness of Our results show that the majority of the designer respon- NUI execution. Approximately one third of the respondents dents are satisfied with their current NUI documentation prac- scored documentation as moderate or of little importance for tices (D1). Nonetheless, this satisfaction is not necessarily correctness. reflected on the quality and extend of applied documentation practices (D2, D3). Those observations unveil that NUI docu- Scientific publications mentation is generally an underestimated or ignored problem Figure 4 illustrates the complete classification of the analyzed by interaction designers and developers. papers based on the previously presented methodology. Pa- Clearly, the NUI paradigm vastly grows in terms of the num- pers that satisfy the conditions are distinguished with a cod- ber of interaction proposed, the diversity of interaction types, involved body parts, involved actions, etc. [3][4][8]. Great advancements, in terms of innovation and usability evalua- tion, of this type of interaction are usually proposed and pre- sented at various venues and conferences such as ACM CHI (Human Factors in Computing Systems) and UIST (User In- terfaces Software and Technology). Despite this effort, some researchers believe that very little effort is actually targeted to improve the reliability of systems offering and adoption these kinds of novel interaction techniques [8]. Soon the lack of adequate interaction documentation and dis- Figure 3. The Designers’ Perceived Importance of Documentation for semination will lead to challenge the design and engineer- Acceptance, Correctness, Experience, and Sharing of NUI ing of interactive systems. Documentation can be used to Figure 4. ACM Multitouch and Motion based Interaction Papers Review - Interaction Documentation Practices and Habits Analysis for The Years 2012 and 2013 (until 22.08.2013) extract information about the type of movements involved the currently applied media such as text, pictures, sketches, in the interaction, involved body parts, adequate interaction and videos may lead easily to losing parts of the movements, execution, etc. The absence of such information will nec- overly complicated descriptions, losing timing information, essarily lead to burden the deployment of interaction tech- etc. niques in automated interactive systems, especially processes In fact, according to Navarre et al. [8], formal interfaces such as context acquisition, reasoning, interaction filtering, etc. are greatly hindered. The absence of documentation in- description languages support interaction at the development evitably challenges analyzing the proposed interaction tech- (e.g., prototyping) as well as the operation phase, while con- niques. For instance, requirement analysis can be greatly ventional empirical or semiformal techniques lack to provide compromised, analyzing physical requirements for NUI users adequate and sufficient insights about the interaction (e.g., is not possible, or correctly reproduce or extend a particu- comparing two design options with respect to the reliability lar interaction techniques becomes an extremely challenging of the human-system cooperation). task. The notion of movement is of particular importance for Ki- netic Interactions, as it resides at the core of this type of inter- Documentation types and methods actions. Movement documentation is a very relevant and gen- The results show that various media types are used by design- erally a very unresting problem for many fields such as dance ers for documentation. From one hand, designers reported choreography, movement rehabilitation, motion recognition that the most used media types for documenting NUI are and analysis, and human movement simulation. Accord- text, pictures, sketches, and videos respectively (D5). The ing to Kahol et al. [7], having such languages and notations academic paper investigation on the other hand shows text, features three main qualities: facilitate teaching and learn- sketches, and pictures respectively used in the analyzed pa- ing of movement styles, permit the writing of universally- pers (P1). Despite the aforementioned difference, it is clearly understood scores of movement, and provide a universal lan- visible that designers rarely use formalized languages or ap- guage to communicate movements. Nonetheless, Kahol et proaches as a documentation medium (D3, D6, P2). Formal- al. [7] still acknowledge the lack of a formalized languages ized description of NUI are an important mean to document and notations of generic motion, matching our investigation various aspects on the interaction to insure integrity and cor- results (especially D4). We share the same viewpoint as in rectness of execution, and the possibility of replicability. [8], lacking adequate and formalized documentation lead in- evitably to increase the gap between the design and (commer- Designers are not in favor of formal languages and audio. cial) deployment of developed interaction techniques. This can be due to the complexity of language learning, and the complexity of describing movements respectively. Whilst formalized languages can be hard to learn and apply, they The importance of documentation are often used in different fields for documenting movements. Designers recognize the importance of documentation for the Formalized languages have a clear benefit to preserve and users’ experience, the acceptance of NUI techniques, and transfer the technique to other designers without endanger- correctness of use. The most important use of documenta- ing the originality and vital aspects of the technique. Using tion is for sharing NUI techniques (D7). Sharing is particu- larly important for different purposes such as communicating interactive systems for large-scale dynamic runtime deploy- NUI to other peer designers, improving NUI functionality by ment of existing and future interaction techniques is greatly other designers, adopting NUI techniques in various interac- challenged. tive eco-systems, and reaching user audience. Even though designers recognized these important roles, their documenta- REFERENCES tion practices appear generally ignorant to this importance. 1. Altakrouri, B., Gröschner, J., and Schrader, A. Documenting natural interactions. In CHI ’13 Extended Documentation challenges in future ambient spaces Abstracts on Human Factors in Computing Systems, CHI So far we have discussed the current NUI documentation EA ’13, ACM (New York, NY, USA, 2013), 1173–1178. practices, but the shift towards future ambient spaces imposes new requirements, and challenges the current practices. 2. Altakrouri, B., and Schrader, A. Towards dynamic natural interaction ensembles. In Fourth International This type of interactive systems aims at avoiding mismatch Workshop on Physicality (Physicality 2012) co-located problems between user’s needs and device’s offers, by em- with British HCI 2012 conference, A. D. Devina ploying the best matching interactions to the given context, Ramduny-Ellis and S. Gill, Eds. (Birmingham, UK, 09 hence the user independence (acceptability by permitting cus- 2012). tomizability) and usability qualities required by Wachs et al. [10] are inherently enhanced. Pruvost et al. [9] noted that 3. England, D. Whole body interactions: An introduction. interaction environments are becoming increasingly hetero- In Whole Body Interaction, D. England, Ed. Springer geneous and dynamic, hence they are no longer static and London, 2011, ch. Whole Body Interactions: An closed; the interaction context is becoming increasingly more Introduction, 1–5. complex; and, increasing adaptability is required for sustain- 4. Fogtmann, M. H., Fritsch, J., and Kortbek, K. J. able utility and usability. Kinesthetic interaction: revealing the bodily potential in Current NUI documentation practices, as discussed in this pa- interaction design. In Proceedings of the 20th per, are greatly challenged by such a system. The current Australasian Conference on Computer-Human documentation practices and strategies are not adequate and Interaction: Designing for Habitus and Habitat, OZCHI fail to meet the challenge of dynamically created documen- ’08, ACM (New York, NY, USA, 2008), 89–96. tation for interaction ensembles. Interactions are currently 5. Hamon, A., Palanque, P., Silva, J. L., Deleris, Y., and ego-centric and designed in isolations, so is the documenta- Barboni, E. Formal description of multi-touch tion. Such isolation implies a complete absence of informa- interactions. In Proceedings of the 5th ACM SIGCHI tion about the interaction’s behavior as part of an ensemble in Symposium on Engineering Interactive Computing a dynamically changing eco-system. Systems, EICS ’13, ACM (New York, NY, USA, 2013), 207–216. FUTURE WORK As part of our research roadmap, we will continue to ex- 6. Iacolina, S., Lai, A., Soro, A., and Scateni, R. Natural plore this field by (1) extending our investigation to study the interaction and computer graphics applications. In differences and similarities between NUI documentation in Eurographics Italian Chapter Conference, E. Puppo, academic and commercial settings (as in motion-based and A. Brogni, and L. D. Floriani, Eds., Eurographics touch-based application market initiatives); (2) extending our Association (Genova, Italy, 2010), 141–146. analysis to include NUI users and their learning habits; and 7. Kahol, K., Tripathi, P., and Panchanathan, S. (3) extending our ongoing work on a dedicated tool for docu- Documenting motion sequences with a personalized menting NUI annotation system. IEEE MultiMedia 13, 1 (2006), CONCLUSION 37–45. We have presented an investigation on the applied practices 8. Navarre, D., Palanque, P., Ladry, J.-F., and Barboni, E. and habits to document and share developed interaction tech- Icos: A model-based user interface description niques. The analysis included: (1) an online exploratory sur- technique dedicated to interactive systems addressing vey on documenting Natural User Interfaces (NUI) answered usability, reliability and scalability. ACM Trans. by 64 designer; and, (2) coding and analyzing a sample of Comput.-Hum. Interact. 16, 4 (Nov. 2009), 18:1–18:56. 93 recently ACM published multitouch and motion-based in- teraction papers. Our study reveals that good documentation 9. Pruvost, G., Heinroth, T., Bellik, Y., and Minker, W. practices are rare and largely compromised. Our survey re- User Interaction Adaptation within Ambient veals that there is a lack of documentation tools, methods, and Environments, next generation intelligent environments: formal languages; designers almost never follow or apply any ambient adaptive systems ed. Springer, Boston (USA), documentation standards; and designers never use available 2011, ch. 5, 153–194. interaction authoring tools. Hence, the creation of a collec- 10. Wachs, J. P., Kölsch, M., Stern, H., and Edan, Y. tive long lasting interaction heritage remains unachievable. Vision-based hand-gesture applications. Commun. ACM Moreover, the gap between developing and rightly dissem- 54 (February 2011), 60–71. inating interaction techniques increases. Thus, engineering