-
INTRODUCTION
As a result of the ongoing research and development in the
areas of ubiquitous and pervasive computing the variety of
heterogenous commercial devices with rich functionalities and
novel interaction techniques arise. Particularly with regard
to the fields of Ambient Assisted Living (AAL), Internet of
Things (IoT) and Smart Home, users are increasingly faced
with natural interaction. While bulk of HCI research strives
to create interaction techniques that are easy to learn,
natural, self-explaining, and novel, documentation of interaction
techniques is generally an underestimated and ignored issue
or simply considered luxury and unnecessary. Considering
the progressive complexity in ambient scenarios containing
heterogenous devices and interaction techniques, this results
Workshop on Large-scale and model-based Interactive Systems: Approaches and
Challenges, June 23 2015, Duisburg, Germany.
Copyright c 2015 for the individual papers by the papers’ authors. Copying
permitted only for private and academic purposes. This volume is published and
copyrighted by its editors.
into an increasing gap between the users’ ability to learn and
remember these techniques and the provided functionality.
Currently, documentation of smart devices’ interaction
techniques in ambient space scenarios is usually spatial
distributed and highly eco-centric, if accessible at all.
Concerning the ongoing interconnection of devices and interaction
behavior in terms of IoT, such documentation can’t be
realized manually. In total, the variety of current Smart Objects
challenge users in accessing and operating. It is reasonably
assumed, that these interaction challenges will increase
significantly caused by complexity and unpredictable
interconnections in ambient spaces.
SMART AMBIENT SPACES
Ambient spaces are manifested by an expanding world of
interconnected Smart Objects full of rich interaction
capabilities driven by ubiquitous and pervasive technologies.
Research and industrial development in this area have resulted
into vast increase in the number of smart commodity devices
and objects (thereafter, called Smart Objects) seamlessly
interweaving in a wide range of inhabited environments (e.g.,
households). A recent study conducted by BITKOM
(Federal Association for Information Technology,
Telecommunications and New Media) revealed that every household in
Germany owns at least 50 electrical devices with an
increasing tendency towards more devices [
4
] and half of all
household devices are expected to be connected as part of a network
by 2018 as reported by RWE Effizienz GmbH [
25
].
While users are currently familiar with handling normal
physical objects and with interacting with simple and often limited
number of electrical devices [
25
], the variety and diversity
of functions and handlings of Smart Objects pose new
challenges, especially to enable and familiarize users with
interaction possibilities in ambient spaces [
5, 28
].
The increasing number of devices as well as the increasing
diversity of offered functions imposes serious learning issues
for the user according to Poppe et al. [
20
]. In one of his
articles, Norman argued that this may easily lead to long-term
usability obstacles and inflate problematic and irrational use
of devices [
16
]. For instance, in case of a time change,
different clocks in households offer inconsistent ways and
interaction modalities for modifying the time. Hence, even this
simple operation normally challenges the user [
11
]. Such
challenges easily evolve with more emphasis on the required
implicit knowledge of users and the lack of adequate
documentation [
30
]. Sometimes devices can’t even be controlled
without the use of additional material [
27
].
The interaction challenges and difficulties with current and
future smart devices and artifacts were also the subject of
Norman’s book, titled Living with Complexity [
17
]. In this
book, Norman drew a clear distinction between complexity
and complication. While complexity refers to the form of
presentation of possible interaction states and transitions,
complication donates the psychological state of a person who tries
to learn an interaction with an object. Hence, complex
objects and artifacts are not necessarily complicated to interact
with. Complication barriers can appear due to different
reasons including changing the environment and simultaneously
changing artifacts. We believe that ambient spaces may result
into various complication barriers due to three inherited
characteristics reported by Pruvost et al. [
23
], namely, the
heterogeneity and distributivity (containing a variety of devices with
various capabilities); dynamic media mobility (interaction
capabilities are highly dynamic as interaction devices may join
and leave the ambient space at any time), and user mobility
(challenging users to attend to interaction needs). This leads,
very commonly, to missing the natural mapping of offered
functionality and adequate interaction modalities [15, p. 12],
as well as to hindering the user from building the correct
mental model of the system.
OPPORTUNITIES AND
CHALLENGES
OF NATURAL
USER INTERFACES
Recent advancements in HCI research have revealed new and
novel interaction techniques to operate and control devices in
ambient spaces by using Natural User Interfaces (NUIs) as
in multi-touch gestures, motion-gestures, gaze-interactions,
etc. [
8
]. In literature, different definitions of interactions with
NUIs were elaborated and most of them refer to the user’s
natural abilities, practices, and activities to control interactive
systems. Many of those interactions are mostly caused and
characterized by motion and movement activities, ranging
from pointing, clicking, grasping, walking, balancing,
dancing, etc. as discussed in [
1
].
In the last 10 years, NUIs, using touch and motion
enabled technologies, found their way commercially and
became widely accessible to the end user. Moreover, users are
becoming more acquainted with using different body parts
to interact with applications such as gaming (e.g.,
motioncontrolled active play by Microsoft Kinect or the Wii system),
data browsing, navigation scenarios (e.g., tilting for scrolling
photos as in iOS and Android devices), and many more. This
has encouraged the HCI community to continuously expand
towards the NUI paradigm and currently various new calls
have been arisen to explore new potential in designing for the
whole body in motion [
7, 9
]. Despite the efforts towards
intuitive and simple interfaces, the NUI paradigm is challenged
by an expanding user population and diversity with respect
to age and physical abilities, as discussed in [
1
]. On the one
hand, the naturalness of NUI does not imply the simplicity
to recall and use interaction techniques [
18
]. On the other
hand, utilizing the human body and its parts for interaction
comes with its own set of complexities. Simple commands,
like ”raise your arm”, may have very different interpretations.
Different aspects are important to consider for correctly
executing such a simple command, for instance movement
direction, involved body parts, timing information, etc.
GUIDANCE IN AMBIENT SPACES
In order to correctly use simple or complex technologies, the
availability and accessibility of relevant information are
essential for the user. Therefore, Norman [
14
] coined the term
affordances in respect of objects’ self-revealing interaction
possibilities to easily enable users interacting with them. The
same applies for the interaction in ambient spaces, however
the current concept of affordances does not apply to the
ongoing embedding of devices and accordingly their
interaction possibilities [
26, 22
]. The dynamic nature of ambient
spaces imposes different learning and affordance challenges
on users. In this regard, relaying solely on visual appearance
and affordances of a smart object to explain its logic and
function are not enough [
26
]. Hence, adequate documentation and
presentation of interaction possibilities and the utility of an
object are essential part for learning ambient spaces, which
aim at correct usage of devices and optimized user mental
models.
In ambient spaces, documentation is not only vital for the use
of objects but also for the design process itself and for a
successful share and exchange of components and knowledge.
Although different device manufacturers pay attention to the
consistency of interaction patterns and product descriptions,
there are currently no consistent and unified standards for
describing smart objects and their offered interaction
possibilities in ambient spaces. Based on this, users have to
repeatedly remember how to interact with such devices [
24
]. This
recurring state of knowledge between beginner and expert in
interacting with a device is called perpetual intermediate [6,
p. 42].
Our previous work on reviewing existing
documentationrelated tools for NUIs revealed four general observations or
shortcomings, namely the lack of widely adopted tools by
NUI designers, the absence of dedicated NUI documentation
tools, the lack of end-user support, and the lack of support
and considerations of body movements and postures as part of
the interaction descriptions (if at all found). Furthermore, the
review revealed that there is a lack of formalized languages
and notations of generic motion documentation [
1, 3
]. For
the previously mentioned reasons and potentially more,
people turn to rely on other learning approaches and methods.
Trial-and-Error is a very common practice to unveil adequate
system interactions used by users. However, it is not
necessarily the most effective approach in many cases. This was
the subject of many research studies in the area of safety and
critical environments. A study has revealed that 70% of
surgeons and 50% of nurses demonstrated problems dealing with
medical devices in operating theaters [
12
], where 40% of the
respondents indicated that the ignorance of adequate
operation guidelines of medical devices have resulted into
repeatedly occurring hazards. In a previous study [
1
], the majority
of reported respondents of a questionnaire (more than 90%)
rely on try and error to learn interactive techniques of
personal interactive devices (e.g., smart phones, interactive TVs,
handhelds, and game consoles). This can be due to the limited
range and simplicity of interaction features currently
available in the users’ commodity devices (e.g., swipe, shake, and
pan). Nonetheless, there is a strong evidence that learning and
memorizing interaction techniques will become more
complex due to the vast growth of multi-touch- and motion-based
interactions in terms of, but not limited to, the number of
interactions proposed, the increasing complexity of interaction
techniques, expanding diversity of interaction types, involved
body parts, involved actions, and runtime ensembles of
interaction techniques [
7, 9, 13, 2
]. This clearly advocates the
need for reference documentation of interaction techniques as
a necessity and an aid tool for users [
19
]. In fact,
interactivity in ambient spaces is becoming even increasingly dynamic
(interaction environments are becoming increasingly
heterogeneous and dynamic and no longer static and closed [
23
]),
adaptive (required for sustainable utility and usability), and
multi-modal. Hence, interactive ambient spaces are created
in an ad-hoc manner, where multiple interaction techniques
grouped together to adapt the available interaction resources
and possibilities to the user’s physical context and abilities.
This shift towards an evolving world of interactivity (smart
spaces, user mobility, anthropomorphic abilities and
disabilities, preferences, etc.) requires new dissemination,
deployment, and adaptation mechanisms for NUI. For these
reasons, documentation for training, demonstration, and
reference purposes plays a major role to set the limits and
boundaries for NUI deployment and adoption in interactive ambient
spaces.
A FRAMEWORK FOR AMBIENT REFLECTION
In order to offer a possibility to compensate the previously
mentioned emerging problems in interaction and
documentation, we strive for developing a three-divided framework
for Ambient Reflection as an integral component of reflective
systems self-x properties [21, p. 322 et seq.]. By providing
this framework as a feasible solution, we foster the
multimodal self-description of (interconnected) devices regarding
interaction possibilities. In total, our envisioned framework
consists of three building blocks, namely an Ambient
Reflective Documentation Language, Documentation Fusion and
Presentation Oriented Publishing. In the following
paragraphs, these components are described in detail.
Ambient Reflective Documentation Language
Current possibilities for technical documentation are limited
to unstandardized media entities, i.e., each device is described
in different modalities using different types of media in
various formats. Hence, caused by this diversity an automated
processing is not possible. In order to achieve this
property a unified extensible documentation language for
ambient spaces should be provided, covering a structured
description about devices specification and interaction possibilities
on a high granularity (further referred as micro-level).
Moreover, a documentations content should be decoupled from its
presentation in order to achieve more flexibility for further
processing, which already has successfully been done (e.g.
by [
29
]). This approach of presentation-neutral describing of
Smart Objects may guarantee distributivity, extensibility and
further presentation oriented processing.
Documentation Fusion
Given the assumption, that relevant devices in an ambient
space scenario were described by an ambient reflective
documentation, the documentation fusion will take place. In the
following, a device with access to its remote located or
attached documentation is called documentation entity.
Including the current context and environmental state, the fusion
step performs in-situ processing and merging of distributed
documentation entities. As result it generates a
presentationneutral adaptive ambient space manual for interconnected
ensembles of Smart Objects in ubiquitous and pervasive
environments, considering just involved devices and interaction
possibilities.
Distributing the generated material to a dedicated
coordinating engine will enable a guidance system to offer further
instructions regarding interaction of device ensembles at the
time the user needs or asks for support. Nonetheless, the
fusion step might be skipped to provide even single device
interaction guidance as well.
Presentation Oriented Publishing
Using concepts of presentation oriented publishing for
markup languages [
10
] adds an additional abstraction layer
between the generated manual and the final presentation of
instructions to the user. The formerly mentioned decoupling
properties of presentation-neutral documentation facilitates
the documentation language translation into other renderable
languages. Furthermore, the inclusion of a standardized style
description to the translated language concerning
presentational aspects will offer a possible solution for adaptivity in
presentation. Finally, the fused documentation should be
deployed to appropriate rendering devices. E.g. a fused
documentation might be translated into the Scalable Vector
Graphics format using different color sets for color-blind users and
visualized by an internet browser’s rendering engine.
EXEMPLARY SCENARIO
In order to further illustrate the frameworks’ working-process
the following scenario is assumed (see figure 1): A user
resides in an environment containing of n Smart Objects. Since
a device ensemble, consisting of Smart Object 1 and 2, was
built, the user needs guidance in usage. Object 1 is
already described by using the Ambient Reflective
Documentation Language, where Object 2 still needs to be described.
Therefore, an adapter is used to translate existing
documentation, written by the manufacturer, into the documentation
language. It is likewise conceivable, that an ambient
reflective documentation is remote located, whereas the device just
provides the destination (as done by Smart Object n). The
Fusion Engine fetches and merges the documentation of the
involved Objects as well as applying a stylesheet based on the
users preferences. Finally, the generated manual is deployed
to rendering devices 2 and k and by association delivered to
the user. It should be noted, that a rendering device might be
equal to a Smart Object in the environment and thus might
also be documented. Hence, a set of documented rendering
devices might form a device ensemble with other Smart
Objects and thus are by definition documentation entities.
CONCLUSION
Considering the current development in the areas of IoT, AAL
and Smart Home face users with new challenges in terms of
Human Computer Interaction. Devices’ and their
functionality are progressively interconnected, embedded and new
interaction techniques within the scope of Natural Interaction
arise. As a result of the ongoing disappearing of user
interfaces as well as the emerging usage of gesture control, the
current concept of affordances may not apply to current
developments. Existing difficulties in HCI will increase in the
areas of ubiquitous and pervasive computing, caused by the
environments high complexity, heterogeneity and dynamic.
Beyond this, present technical documentation does not
follow a common pattern or is adapted to the user’s needs. As a
possible solution to tackle these problems, we presented the
approach of a conceptual ambient reflection framework,
consisting of three major components: An Ambient Reflection
Documentation Language for describing interaction
possibilities of Smart Objects on a micro-level, documentation fusion
using the description language to merge documentation
entities of interconnected devices for generating an ambient
manual tailored to the users’ context and needs as well as the
presentation oriented publishing for multi-modal rendering the
manual in-situ. In total, we strongly believe to counteract
emerging interaction problems in ambient space scenarios by
further investigating this framework.
FUTURE WORK
Next, we take to carry out a study to identify different
contexts and therefore needs of users with respect of
documentation and guidance in interaction. Including these findings
and further research regarding description languages, we will
develop a unified Ambient Reflective Description Language
for Smart Objects and apply it to a representative set of Smart
devices, composed of different device categories. In addition,
we try to determine a set of generic rules and processes in
order to achieve a consistent manual generated by the fusion
engine. Upon this, the development of interweaving style
description and documentation and the delivery to rendering
devices should enable Smart Objects to describe themselves.
Finally, we plan to evaluate our framework by carrying out
a scenario-based evaluation to determine the precision of the
fusion itself as well as the usefulness of our provided
guidance for the user.