A mixed scenario involves a lot of objects which may be related in various ways. These relations may lead to inconsistencies similar to those related to continuous interaction. We propose here a model for the declarative representation of the design aspects involved in a MIS (Mixed Interaction Space). User-interface design, mixed interaction space, spatial integration, temporal integration, mixed reality systems. Mixed Reality (MR) is the state-of-the-art technology that merges the real and virtual worlds seamlessly in real time. It draws attention as a new technology of human interface, which surpasses the border that the conventional virtual reality has.
In view of the multidisciplinary integration and associated
complexity existing in MR systems, the reality paradigm
given by [
The user’s interaction with this Reality-Virtuality
Continuum can be augmented by tangible interface.
According to [
Bâtiment Stévin, Place du Levant, 2
As example of how uses the approach for designing Mixed
Interaction Space we will take account the Image-guided
surgery (IGS) interaction space scenario. In such systems
complex surgical procedure can be navigated visually with
great precision by overlaying on an image of the patient a
color coded preoperative plan specifying details such as the
locations of incisions, areas to be avoided and the diseased
tissue. It is a typical application of augmented reality (AR)
systems where the virtual world corresponding to the
preoperative information should be correctly aligned in real
time with the real world corresponding to the
intraoperative information. This study case was thoroughly
discussed by the authors in [
Note that the terms ‘digital’ and ‘virtual’ are used in this work in the sense of not physical or real.
The terms “real” and “physical” are used in the sense of not digital or virtual.
An Interaction Space (IS) is assumed to be the complete
presentation environment required for carrying out a
particular interactive task. And it requires very often to deal
with questions such as whether particular objects or scenes
being displayed are real or virtual, whether images of
scanned data should be considered real or virtual, whether a
real object must look UHDOLVWLF whereas a virtual one need
not to, etc. For example, in some AR systems there is little
difficulty in labeling the remotely viewed video scene as
UHDO and the computer generated images as YLUWXDO. If we
compare this instance, furthermore, to a MR system in
which one must reach into a computer generated scene with
one's own hand and "grab" an object, there is also no doubt,
in this case, that the object being grabbed is "virtual" and
the hand is "real". Nevertheless, in comparing these two
examples, it is clear that the reality of one's own hand and
the reality of a video image are quite different, suggesting
that a decision must be made about whether using the
identical term UHDO for both cases is indeed appropriate.
In this work we adopt the distinction given by [
Now we can say that an interaction space is composed of: x x x
Real Interaction Space (5,6): if and only if it is composed of real components, e.g. real concrete interaction objects such as physical objects.
Virtual Interaction Space (9,6): if and only if it is composed of virtual concrete interaction objects; Mixed Interaction Space (0,6): if and only if it is composed of virtual concrete interaction objects added to the real environment e.g. combined with real concrete interaction objects.
and of a 5HDO ,QWHUDFWLRQ 6SDFH 5,6 , which are supposed to be physically constrained by the XVHU V ZRUNVSDFH and which may be all displayed on the workspace simultaneously.
Each ZRUNVSDFH is composed of at least one ,QWHUDFWLRQ 6SDFH (,6) called the basic IS, from which it is possible to derive the other IS (Figure 1). This configuration becomes needed once the user can manipulate objects in the virtual world through the VIS or the user can manipulate objects in the real world through the RIS. &RQFUHWH ,QWHUDFWLRQ 2EMHFW (CIO): this is an object belonging to the Interaction Space that any user can see with the appropriate artefacts (e.g. See-through head mounted display). We have two types of CIO, real and virtual. The Real Concrete Interaction Object is part of the RIS (e.g., live video, some physical objects like a pen, a needle, which can have a representation in the virtual world and so it will become a virtual concrete interaction object (Figure 1). The Virtual CIO is a part of the VIS (e.g. text, image, animation, push button, a list box). The virtual CIO can also entail the virtual representation of the real CIO. A CIO is said to be VLPSOH if it cannot be decomposed into smaller CIOs. A CIO is said to be FRPSRVLWH if it can be decomposed into smaller units. Two categories are distinguished: SUHVHQWDWLRQ &,2, which is any static CIO allowing no user interaction, and FRQWURO &,2, which support some interaction or user interface control by the user. Both, presentation and control CIOs can be part of the RIS and/or VIS. $EVWUDFW ,QWHUDFWLRQ 2EMHFW (AIO): this consists of an abstraction of all CIOs from both presentation and behavioral viewpoints that is independent of any given computing platform. By definition, an AIO does not have any graphical appearance, but each AIO is connected to 0, 1 or many CIOs having different names and presentations in various computing platforms.
Regarding the vast possibilities to compose, to interact and to insert the Interaction Space into the environment we may to take account the follow design aspects which are described in Figure 2: x spatial integration; x temporal integration; x insertion context; x user’s interaction focus.
is displayed require The interaction space may involve a large number of media objects which should be integrated into the MIU (Mixed Interaction Unit). This integration concerns the spatial ordering and topological features between Concrete Interaction Objects (e.g. all participating visual media objects).
Then in the context of an AR application, a designer would
like to place spatial objects (text, images, videos, animation,
etc.) in the Interaction space in such a way that their
relationships are clearly defined in a declarative way, i.e.,
“text A is placed at the location (100,100), text B appears 8
cm to the right and 12 cm below the upper side of A”.
As related by [
A N-dimensional projection relation is a N-tuple of 1D relations, e.g. 5 = (5 ,5 ). Each 1D relation corresponds to the relationship between the N-dimensional objects in one of the dimensions. So if V is the number of possible 1D relations at a particular resolution, the number of ND relations that can be defined at the same resolution is V . According to the requirements of the particular application, not all dimensions need to be tuned at the same resolution, in which case the maximum number of ND relations is the product of the corresponding numbers for each dimension. Figure 3 illustrates the 169 (132) primitive projection relations between regions on the plane, on the initially discussed (Allen's) resolution scheme. All previous properties can be analogously extended to N dimensions.
So, given a N-dimensional relation, the corresponding
spatial configuration can be easily inferred by combining all
the 1D configurational inferences. The complete description
of this approach can be found in [
To specify the spatial integration we propose to use the
generalized methodology for representing the distance
between two spatial objects, given in [
The distance will be expressed in terms of distance between the FORVHVW YHUWLFHV. For each spatial object O, we label its vertices as 2 YL L starting from the bottom left vertex in a clockwise manner. As FORVHVW, we define the pair of vertices ($ YL % YM ) with the minimum Euclidean distance. The designer of a mixed interaction space must be able to express spatial composition predicates in an unlimited manner. For instance (see Figure 4), the designer could describe the appearing composition as: “REMHFW % WR
So, assuming two spatial objects $ %, we define the generalized spatial relationship between these objects as:
identifier of the topological-directional relationship
between $ and % (derived from [
³7KH ,*6 LQWHUDFWLRQ VSDFH VWDUWV ZLWK EDFNJURXQG
Ri_13
The real scenario of this description is illustrated in Figure 5 and the spatial composition (interaction space layout) of the above scenario is illustrated in Figure 6, while the temporal one will be discussed in the next sub-section. The directional and relational relationships between the objects in many applications of augmented reality result from the registration procedures to mix in the correct way real and digital worlds. For instance, the AR systems which are based on markers recognition in order to relate the real and virtual worlds (such as those using ARToolKit1 library), assume that the marker is in x-y plane, and z axis is pointing downwards from the marker plane. So, vertex positions can be represented in 2D coordinates by ignoring the z axis information and then the virtual object can be placed in a (x, y, z) position related to the center of the marker.
These spatial aspects can be defined by: 1. designer (while design time), 2. by user 3. or by the system (while the application progresses). This classification will be used as a VSDWLDOBFRQWUROB,' parameter in the composition of mixed interaction spaces. The spatial integration of objects into the interaction space is a relevant aspect since that information facilitates processing through efficient allocation of attentional 1 More information about ARToolKit can be found at http://www.hitl.washington.edu/research/shared_space/dow nload/ resources. For instance an adequate spatial integration of the objects can facilitate the user’s interpretation. the tracking system and they disappear according to user’s interaction.
Besides the spatial aspects related to the integration of CIO into MIU we should also consider the temporal aspects that involve all participating media objects (e.g. visual and sound).
As mentioned in [
3. Third part (e.g. defined by an agent system which is capable of making decisions and initiating actions during execution time independently) This classification will be used as a WHPSRUDOBFRQWUROB,' parameter in the composition of mixed interaction spaces. Figure 7 shows the temporal synchronization diagram related to the spatial diagram illustrated in Figure 6. The text objects C, D and E appear automatically according to 2 3 4 5 6 7 9 10 11 12 13 Besides spatial and temporal integration of interaction space objects it is important to understand how the insertion of devices and interaction spaces in the environment can contribute to a better interaction.
According to the user’s focus while performing a task we have identified four spatial zones for an insertion device considering the level of periphery (see Figure 8): 1. Central zone: it corresponds to a device insertion distance of 0 to 45cm from the user’s task focus. 2. Personal zone: it corresponds to a device insertion distance of 46cm to 1.2m from the user’s task focus. 3. Social zone: it corresponds to a device insertion distance of 1.3 to 3.6m from the user’s task focus. 4. Public zone: it corresponds to a device insertion distance bigger than 3.6m from the user’s task focus. The four possible insertion context type discussed here will be used as ,QVHUWLRQB&RQWH[WB,' parameter in the composition tuple of mixed interaction spaces.
Figure 8. Zones of insertion context according to user’s task focus. 1.Central zone; 2.Personal zone; 3.Social zone and 4.Public zone.
If the device is inserted in the central zone of the user’s task, s/he does not need to change her/his attention focus to perform the task. Otherwise if the user is changing the attention focus all time, then in this case it is probable that the device is inserted outside from the central zone and so in a peripheral context of use (Figure 9).
In the Museum project, one application of NaviCam system
[
For example in the Museum project [
Following the definition given by [
Therefore, by considering all possibilities of interaction focus while the user is performing a specific task, we have found five possible combinations:
Interaction focus on Real World without shared attention (RW): in this case the interaction is focused on only one item in the real world. There are no real items competing for user’s attention.
Interaction focus on Virtual World without shared attention (VW): in this case the interaction is focused on only one item in the virtual world. There are no virtual items competing for user’s attention.
Interaction focus Shared in the Real World (intra-world interaction focus, SRW): in this case the interaction focus is shared between items in the real world.
Interaction focus Shared in the Virtual World (intraworld interaction focus, SVW): in this case the interaction focus is shared between items in the virtual world.
Interaction focus Shared between Worlds (inter-world interaction focus, SW): in this case the interaction focus is shared between items belong to different worlds (real and virtual).
The five possible interaction focus types discussed here will be used as ,QWHUDFWLRQB)RFXVB,' parameter in the composition tuple of mixed interaction spaces.
This declarative definition should be transformed into an internal representation that captures the topological, directional, temporal relationships as well user’s interaction focus and insertion context of IS. Here we propose a definition model to support these needs.
Then the composition of a mixed interaction space consists of several LQGHSHQGHQW fundamental compositions. The term LQGHSHQGHQW implies that objects participating in these compositions are not related implicitly (either spatially, or temporally, or by interaction focus or insertion context), except for their implicit relationship at the start point .
Thus, all compositions are explicitly related to . We call these compositions FRPSRVLWLRQ WXSOHV, and these include spatially and/or temporally related objects.
MIS composition = {[6SDWLDOB,QWHJUDWLRQ], [7HPSRUDOBLQWHJUDWLRQ], [,QWHUDFWLRQB)RFXV], [,QVHUWLRQB&RQWH[W]} contains the following optional
parameters: >6SDWLDOB,QWHJUDWLRQ@ UHODWLRQBW\SHB,' 9
VSDWLDOBFRQWUROB,'
9
5HODWLRQB7\SHB,' is given by one of the possible
relationships presented in [
The 7HPSRUDOB,QWHJUDWLRQ can have the following optional parameters: 9 DQG 9 are the closest vertices between two objects $ and %, respectively, and [ \ are the horizontal and vertical distances between YL YM. 5HODWLRQBW\SHB,' is given by one of the Allen’s relations ID represented in Table 1. ,QVHUWLRQB&RQWH[W corresponds to the insertion context of the interaction space into the environment. This is a parameter defined only for the main interaction space composition. It can assume one of the 4 possible values discussed in the previous subsection. ,QWHUDFWLRQB)RFXV corresponds to the user’s interaction focus parameter during an interaction. This parameter is defined for each composition and it can assume one of the 5 possible values discussed in the previous subsection; 7HPSRUDOBFRQWUROB,' represents who has the temporal control: user, systems or third part, respectively.
sub-parameters, then: ( The objects to be included in a composition tuple of a MIS are those that are spatially and/or temporally and/or focus shared related. In our example (Figure 6 with spatial integration description and Figure 7 with related temporal integration description) A and B and C should be in the same composition tuple, since A relates to B and B relates to Menu_options. On the other hand, if an object is not related to any other object, neither spatially nor temporally, so it composes a different tuple. The above specifications defined in a high-level are transformed into the following model language considering our example of IGS_InteractionSpace composition:
F > 5 B
B 9 B$ 5: ,*6B,QWHUDFWLRQ6SDFH > 5 B
% > 5 B 9 B% 9 B0HQX
> 5 B B 9 B& B
' > 5 B
,W LV LPSRUWDQW WR VWUHVV WKDW in composition tuple c3 represents the spatio-temporal origin of the Menu_options. In this example, we have a composition of MIS (mixed interaction space). It has to be stressed that, when the host MIS (i.e., IGS_interactionSpace) ends, all the MIS started by it are also stopped (i.e., Menu_options). There is an issue regarding the mapping of the spatio-temporal specifications into the composition tuples: the classification of involved objects. The proposed procedure is the following: For each object $L, we check whether it is related to objects already classified in an existing tuple. If the answer is positive, Ai is classified in the appropriate composition tuple (a procedure that possibly leads to reorganization of the tuples). Otherwise, a new composition tuple, composed by and $L, is created.
During the application development process, it is probable
(especially in the case of complex and large applications)
that authors would need information related to these
relationships. The related queries depending on the spatial,
temporal, interaction focus and insertion context
relationships maybe be classified in the following queries
categories:
x pure spatial or temporal query: only a temporal or a
spatial relationship is involved in the query. For instance,
“which objects always overlap the presentation of live
video A?”, “which objects spatially lie above object B in
the interaction space?”.
x spatio-temporal query: where such a relationship is
involved. For instance, “which objects spatially overlap
with object A during its presentation?”.
x MIS query: spatial or temporal layouts of the application
considering interaction focus and insertion context. For
instance, “what is the spatial integration (layout of MIS)
when the user’s interaction focus is shared between A
and B?”, “which objects are presented when the user’s
focus interaction is focused on the real world?”, “when
the user’s focus is on the real world how is the insertion
context of MIS?”, “when the user has the temporal
control of presentation where is located the user’s
interaction focus?”
The answers of such queries may indicate the potential
problems during interaction such as discontinuous
interaction. For instance if the user has the temporal control
during an interaction and his interaction focus is under
some object in the real world, so he/she probably will
change between operation modes and attention focus to
control, or to interact with the presentation. It characterizes
a functional and perceptive discontinuity during interaction
conforming discussed in [
In this work we have reviewed and extended some approaches to design mixed interaction spaces. With that we have predictively modeled user interaction to evaluate design strategies and support adaptation for continuous interaction while dealing with mixed spaces of interaction. As contributions of this work we have highlighted: x Manage large number of options for the MIS design under development of MR systems. x Acquire spatial, temporal and focused layouts of the MIS under development of MR system for verification purposes such as those related to continuous interaction. x Help designers to envision future interactive mixed systems.
Finally we should be aware that specific design aspects such as spatial and temporal integration of different media objects have implications for the human perception. However the information that people assimilate from a modality of interaction (e.g., visual modality) also depends on their internal motivation, what they want to find and how well they know the domain.
We gratefully acknowledge the support from the Région Wallonne under contract WALEO 21/5129. The work described here is a part of the MERCATOR project available on http://www.tele.ucl.ac.be/PROJ/MERCATOR_MULTI_e.h tml