=Paper=
{{Paper
|id=Vol-91/paper-2
|storemode=property
|title=Generic Interaction Techniques For Mobile Collaborative Mixed Reality Systems
|pdfUrl=https://ceur-ws.org/Vol-91/paperD2.pdf
|volume=Vol-91
|dblpUrl=https://dblp.org/rec/conf/mixer/NigayRBP04
}}
==Generic Interaction Techniques For Mobile Collaborative Mixed Reality Systems==
https://ceur-ws.org/Vol-91/paperD2.pdf
GENERIC INTERACTION TECHNIQUES FOR MOBILE
COLLABORATIVE MIXED REALITY SYSTEMS
Philippe Renevier, Laurence Nigay, Laurence Pasqualetti
Jullien Bouchet
Laboratoire CLIPS-IMAG FT R&D-DIH/UCE
BP 53 38-40 rue G. Leclerc
38041 Grenoble cedex 9 92794 Issy-lesMoulineaux
{renevier, nigay, bouchet}@imag.fr laurence.pasqualetti@francetelecom.fr
ABSTRACT mobile user in the context of a collaborative situation. We
The main characteristic of a mobile collaborative Mixed illustrate the generic nature of the techniques with two
Reality (MR) system is that augmentation of the physical systems that we developed: MAGIC for archaeological
environment of one user occurs through available fieldwork and TROC a mobile collaborative game.
knowledge of where the user is and what the other users are MOBILE COLLABORATIVE MR SYSTEMS
doing. Links between the physical and digital worlds are A mobile collaborative MR system combines the
no longer static but dynamically defined by users to create characteristics of a mobile MR system and of a
a collaborative augmented environment. In this article we collaborative MR system. First a mobile MR system is
present generic interaction techniques for smoothly one in which augmentation occurs through available
combining the physical and digital worlds of a mobile user knowledge of where the user is (the user's location and
in the context of a collaborative situation. We illustrate the therefore the surrounding environment). Second a
generic nature of the techniques with two systems that we collaborative MR system is one in which augmentation of
developed: MAGIC for archaeological fieldwork and TROC the physical environment of one user occurs through the
a mobile collaborative game. actions of other users and no longer relies on information
Keywords pre-stored by the computer. Links between the physical and
Mixed Reality, Mobile Computing, CSCW, Interaction digital worlds are therefore dynamic, based on the users'
Techniques actions. Combining the characteristics of a mobile MR
system and of a collaborative MR system, a mobile and
INTRODUCTION collaborative MR system is one in which augmentation
Mixed Reality (MR) systems seek to smoothly link the occurs through available knowledge of where the user is
physical and data processing environments. This is also the and what the other users are doing.
objective of other innovative interaction paradigms such as
Ubiquitous Computing, Tangible Bits, Pervasive Few MR systems combine the mobile and collaborative
Computing and Traversable Interfaces. These examples of aspects. The main application domain of such systems is
interaction paradigms are all based on the manipulation of game and one of our developed system, TROC, is a game.
objects of the physical environment [2]. Typically, objects Indeed, instead of recreating a virtual world, the existing
are functionally limited but contextually relevant [7]. The games are based in the real world, the system only adding
challenge thus lies in the design and realization of the the magical possibilities related to the game rules.
fusion of the physical and data processing environments WARPING [9] is one example, but one of the users is not
(hereafter called physical and digital worlds). The object of mobile, since s/he is in front of an augmented desktop.
our study is to address this issue in the context of a ARQuake [10] and Human-Pacman [1] are two additional
collaborative mobility situation. Context detection and examples of games. The users are mobile and they must
mixed reality are then combined in order to create a kill digital enemies (ARQuake) or collect digital cookies
personalized augmented environment. (Human-Pacman). In these two examples, we can
nevertheless notice that the links between the physical and
The structure of the paper is as follows: first, we clarify the digital worlds are predefined (positions of enemies or
notion of mobile collaborative MR systems. Having cookies) and the users can only destroy them, they cannot
defined the goal and challenge of mobile collaborative MR create new “links” such as putting a new cookie in the
systems, we then present generic interaction techniques for game field.
smoothly combining the physical and digital worlds of a
Beyond the HCI classical design approach, mobile
collaborative MR systems make it compulsory to use a
. multidisciplinary design approach that embeds
complementary methods and techniques for the design and
evaluation phases. In [5] we present a scenario-based design
approach for mobile collaborative MR systems. In
particular scenarios enable the description of how the The mobile users manipulate objects that are either digital
system would affect the way mobile users carry out their or physical. Interaction techniques must be designed in
individual and collective activities. Based on the functions order to let them manipulate the two types of objects:
integrated in the so-called "projected scenarios", different physical and digital. For flexibility and fluidity of
interaction techniques can be designed. The interaction interaction, such manipulation is either in the physical
techniques, described in the following section, are generic world or in the digital world. We therefore obtain four
and are those supported by our two mobile collaborative cases, by combining the two types of objects and the two
MR systems: MAGIC dedicated to archaeological worlds: the physical world (i.e., the archaeological field or
fieldwork and TROC a mobile collaborative game. the game ground) and the digital world (i.e., the screen of
the pen computer):
GENERIC INTERACTION TECHNIQUES
In order to explain the generic interaction techniques, we 1. Interaction with a physical object in the digital world:
first describe the underlying hardware platform. This is an Mixed interaction.
assembly of commercial pieces of hardware. The platform 2. Interaction with a digital object in the physical world:
includes a Fujitsu Stylistic pen computer. This pen Mixed interaction.
computer runs under the Windows operating system, with a
3. Interaction with a physical object in the physical world:
Pentium III (450 MHz) and 196 Mb of RAM. The
Interaction purely in the real world.
resolution of the tactile screen is 1024x768 pixels. In order
to establish remote mobile connections, a WaveLan 4. Interaction with a digital object in the digital world:
network by Lucent (11 Mb/s) was added. Connections from Interaction in the digital world (graphical user interface).
the pen computer are possible at about 200 feet around the In [5] we fully describe the four types of interaction. We
network base. The hardware platform also contains a Head- focus here on the interaction techniques corresponding to
Mounted Display (HMD), a SONY LDI D100 BE: its the types (1) and (2). For both cases, passive and active
semi-transparency enables the fusion of computer data interaction techniques are designed. Passive interaction
(opaque pixels) with the real environment (visible via techniques are based on tracking mechanisms (such as
transparent pixels). Secondly, a (D-)GPS is used to locate localization and orientation of the mobile user). With
the users. Finally, capture of the real environment by the passive techniques, the user does not explicitly issue a
computer is achieved by the coupling of a camera and an command to the system as opposed to active interaction
orientation sensor. We first used an absolute orientation techniques that correspond to the case where the user issues
sensor, the magnetometer HMR3000 by Honeywell. We a command to the system, for example a drag&drop of an
now use an intertrax 2 that is more accurate. The camera object.
orientation is therefore known by the system. Indeed the
orientation sensor and the camera are fixed on the HMD, in The two types of mixed interaction ((1) and (2))
between the eyes of the user. The system is then able to respectively imply (i) that physical objects must be
know the position (GPS) and orientation (magnetometer or manageable in the digital world (ii) that digital objects
intertrax) of both the user and the camera. Figure 1 shows a must be manageable in the physical world. To do so we
user, fully equipped: the equipment is quite invasive and designed a generic interaction technique, a gateway that
suffers from a lack of power autonomy. Our goal is to plays the role of a door between the physical and digital
demonstrate the feasibility of our interaction techniques by worlds. As a door belongs to two rooms, the gateway
assembling existing commercial pieces of hardware and not exists in both worlds:
by designing specific hardware out of the context of our – the gateway is an area of the physical world, delimited by
expertise. For a real and long use of the platform in a “real” a rectangle displayed in a semi-transparency Head-Mounted
site, a dedicated hardware platform must clearly be Display (HMD),
designed. – the gateway is a rectangular area in the digital world, on
the pen computer screen as shown in Figure 2a (window
entitled “Head Mounted Display”).
Concretely the gateway is simply a window both displayed
on the HMD (Java JFrame) on top of the physical world
and on the pen computer screen (Java JInternalFrame).
Objects in the gateway are visible on the HMD (i.e., in the
physical world) as well as on the pen computer screen (i.e.,
in the digital world). Based on the gateway, we designed
two interaction techniques, namely the “clickable reality”
and the “augmented field”.
– The “Clickable reality” technique: from the physical
world to the digital world. If the object is physical (1), the
object is transferred to the digital world thanks to the
Fig. 1. A user wearing and holding the hardware camera (fixed on the HMD, between the two eyes of the
platform user). The real environment captured by the camera is
displayed in the gateway window on the pen computer
screen as a background. We allow the user to select or click computer. Coordination between users relies on the map of
on physical objects: we therefore call this technique "the the archaeological site, displayed within a dedicated
clickable reality". Before taking a picture, the camera must window (at the bottom left corner of Figure 2a). For each
be calibrated according to the user's visual field. Using the found object, archaeologists fill a form describing the
stylus on screen, the user then specifies a rectangular zone object, draw some sketches or very precise drawings and
thanks to a magic lens (kind of camera lens). The cursor take pictures using the "clickable reality" technique.
displayed on the pen computer screen is also displayed on Analysis of objects relies on comparisons with known
top of the physical world. The corresponding specified zone objects ("Augmented field" technique) from other
(magic lens), displayed in the gateway window on screen archaeologists or reference manuals (database) and on
and on the HMD, corresponds to the physical object to be discussions with other archaeologists in the site or with a
captured. The picture can then be stored in the shared distant expert. After validation, the object is then added to
database along with the location of the object. Note that the shared database and is visible on the map of each user.
although the user is manipulating a magic lens using the Because a picture is stored along with the location of the
stylus on screen, s/he perceives the results of her/his object, we can restore the picture in its original real context
actions in the physical world. (2D representation). When an archaeologist walks in the
– The “Augmented field” technique: from the digital world site, s/he can see discovered objects removed from the site
to the physical world. If the object is digital (2) dragging it and specified in the database by colleagues ("Augmented
inside the gateway makes it visible in the real world. For field" technique). S/he can then see the object as it was
example the user can drag a drawing or a picture stored in a before being removed from the site. The "augmented field"
database to the gateway window. The picture will technique is particularly useful to see objects belonging to
automatically be displayed on the HMD on top of the a stratum lower than the current one, because by definition
physical world. Moving the picture using the stylus on the the objects have all been removed. The MAGIC system
screen will move the picture on top of the physical world. along with its software architecture is fully described in [8].
This action is for example used if a user wants to compare Although the design is based on task and activity analysis
an object from a database with a physical object in the performed in Alexandria (Egypt), we were not able to
field. Putting them next to each other in the real world will experimentally test MAGIC on a site there. In order to
help their comparison. The motion of a digital object (ex: show the generic aspect of our techniques and also to be
drag and drop on the pen computer) can be viewed by the able to perform experimental tests we developed a second
user without looking at the pen computer screen. This is application, TROC, a collaborative game.
because in using the HMD the user can simultaneously
TROC: a mobile collaborative game
view digital objects and the real world. As for the previous
TROC (barter in French) is a mobile collaborative game.
case (1), although the user is manipulating a digital object,
Each player has to collect a list of digital objects that are
s/he perceives the results of her/his actions in the physical
positioned in the game field at the beginning of the game.
world. Transfer of digital objects to the physical world can
As shown in part B of Figure 2b, the digital objects to be
be explicitly managed by the user by drag and drop (active
collected are animals (cat, gull, etc.). Thanks to the
interaction technique) as explained above or can be
"augmented field" technique, the player while moving
automatic (passive interaction technique). Automatic
discovers the objects. TROC also includes 3D sounds that
transfer is performed by the system based on the current
help the player to find the objects. In addition the player
location of the user. When a user walks in the site, s/he can
can use "magical tools" to locate the objects as well as the
see discovered objects specified by colleagues. The
other players on the map displayed on the pen computer
"augmented field" is an example of asynchronous
(part D of Figure 2b, the round circle specifying the zone of
collaboration.
observation). The player can also specify filters (part A of
These generic interaction techniques are supported by two Figure 2b) so that s/he will only see one kind of digital
mobile collaborative MR systems that we developed: object, in the physical world (the game field) as well as on
MAGIC dedicated to archaeological fieldwork and TROC a the map.
mobile collaborative game.
Digital objects collected by a user are stored in four
SYSTEMS: MAGIC AND TROC physical cubes carried by the player. The content of the four
MAGIC for archaeological fieldwork cubes is displayed on the pen computer (part C of Figure
The design of the MAGIC system is based on a study of 2b) as well as on top of the physical cube (Figure 3)
the tasks of archaeological fieldwork, interviews and recognized by a vision algorithm thanks to the camera fixed
observations in Alexandria (Egypt) [5]. The archaeological on the HMD. To collect a digital object, the player has two
fieldwork in Alexandria is time-constrained because the possibilities: first s/he can use the "clickable reality"
archaeological site must be explored in less than three technique or s/he can present a physical cube to the camera
months (rescue archaeology). Tools that can make such fixed on the HMD while issuing the voice command "take"
fieldwork more efficient are therefore important. This is a (Figure 3). The player can also empty a cube and put back
suitable application domain for mobile collaborative MR on the game field a previously collected digital object
systems because archaeologists work in groups, moving in ("augmented field" technique). This is an example of
a delimited site and requiring collections of data. Figure 2a asynchronous collaboration between players.
presents the graphical user interface of MAGIC on the pen
(a)
(b)
Fig. 2. (a) User interface of MAGIC (b) User interface of TROC
In order to win and collect her/his assigned list of objects, We performed a first set of experimental tests of TROC.
the players must collaborate and exchange collected objects. The primary analysis of the collected data shows that 3D
The game is based on the barter technique. During sounds facilitate the location of digital objects, sound
exchanges, a player can lie saying that s/he has a given being available before the object is visible. In addition, the
object and can also give a trapped object to another player. players underlined the fact that the sound reinforces the link
between the physical and digital worlds, by making digital
objects more real. Moreover, it has been observed that
digital objects, the focus of the players, had a strong characteristics of the links between the two worlds. As a
presence to the point that players forgot the physical starting point to this discussion, in [6] we identified two
obstacles. Players underlined the inconsistency of seeing an axes for characterizing the links between the two worlds:
object through a wall and having to go inside the room to the owner of the link (i.e., he/she who is defining the link:
be able to pick it up. Although such a possibility was the designer, one user, all the users) and their
presented as a magical tool which allows one to see static/dynamic character. The link is static if it has been
through the walls, it confirms the fact that consistency fixed during the design. For example in a computer
must be maintained while combining the physical and assisted surgery system that displays anatomical
digital worlds. The participants also wanted to pick up information on top of the patient’s body, the link between
objects by hand. In particular such behavior has been the digital image (anatomical information) and the physical
observed when the objects were very close to the players object (the patient’s body) is static and fixed by the
and therefore very big. Moreover players had more designer. On the other hand, using MAGIC, the users
difficulties to locate objects in a game field without dynamically define new digital objects that are combined
physical landmarks. Indeed, they adopted an approach of with physical objects. As pointed out in [3], instead of
blind searching, while with physical landmarks they first fixing the relationship between the two worlds during the
located the objects on the map and then went to pick them design, "another strategy is to explicitly give the control to
up. the users, allowing them to define and more importantly,
continue to evolve, the relationship between the physical
and virtual documents". A promising way to let the users
specify such links is through multimodal commands [4].
For example in our TROC system, the player could issue
the voice command "this door is now a trap for others"
while designating a door.
ACKNOWLEDGMENTS
This work is supported by France Telecom R&D, under
contract Houria. Special thanks to G. Serghiou for
reviewing the paper.
REFERENCES
1. A. Cheok, S. Fong, K. Goh, X. Yang, W. Liu, F.
Farbiz, Y. Li, “Human Pacman: A Mobile
Entertainment System with Ubiquitous Computing and
Tangible Interaction over a Wide Outdoor Area”, Proc.
mobile HCI 2003, Springer Verlag LNCS 2795, Udine,
September 2003, pp 209-223.
Fig.3. TROC: the content of a cube is displayed on top 2. H. Ishii, B. Ullmer, “Tangible Bits: Towards Seamless
of the physical cube recognized by a vision algorithm Interfaces between People, Bits and Atoms", Proc.
thanks to the camera fixed on the HMD. Conf. Human Factors in Computing Systems (CHI
'97), ACM, Atlanta, March 1997, pp. 234-241.
CONCLUSION
The generic techniques, "clickable reality" and "augmented 3. W. MacKay, W., “Augmented Reality: Dangerous
field", define a reusable hardware and software platform. Liaisons or the Best of Both Worlds?”, Proc. of
We are currently reusing and extending the platform for DARE’00, ACM Press, Elsinore, April 2000, pp. 170-
new applications: we are developing a system that allows 171.
users to annotate physical locations with digital notes, 4. D. McGee, D., P. Cohen, L., Wu, “Something from
which are then read/remove by other mobile users. nothing: Augmenting a paper-based work practice via
The presented interaction techniques therefore constitute the multimodal interaction”, Proc. of DARE’00, ACM
first bricks of a toolkit for developing mobile collaborative Press, Elsinore, April 2000, pp. 170-171.
MR systems. Reusability of the code and independence of 5. L. Nigay, P. Salembier, T. Marchand, P. Renevier, L.
part of it from the hardware are guaranteed by the software Pasqualetti, “Mobile and Collaborative Augmented
architecture model that we applied for developing the reality: A Scenario Based Design Approach”, Proc.
platform [8]. During the workshop, we would like to Mobile HCI 2002, Springer-Verlag, LNCS 2411, Pisa,
address the issue of toolkit for developing MR systems by 2002, pp. 241-255.
making an inventory of existing interaction techniques. 6. L. Nigay, E. Dubois, P. Renevier, L. Pasqualetti, J.
In addition the collaborative situation of the MR systems Troccaz, “Mixed Systems: Combining Physical and
studied in the paper emphasizes one interesting issue about Digital Worlds”, Proc HCI International 2003,
the links between the real and digital worlds, namely: who Lawrence Erlbaum Associates, Crete, 2003.
is the owner of the link, who is the one who defined it. We 7. D. Norman, “The design of everyday things”, Basic
suggest that during the workshop we further study the Books, September 2002.
8. P. Renevier, L. Nigay, “Mobile Collaborative Conference on Intelligent User Interfaces (IUI 2000),
Augmented Reality: the Augmented Stroll”, Proc. New Orleans, ACM, 2000, pp. 256-259.
Engineering for Human Computer Interaction 10. B. Thomas, B. Close, J. Donoghue, J. Squires, P. De
(EHCI'01), Springer-Verlag, LNCS 2411, Toronto, Bondi, W. Piekarski, “First Person Indoor/Outdoor
2001, pp. 315-334. Augmented Reality Application: ARQuake”, Personal
9. T. Starner, B. Leibe, B. Singletary, J. Pair, “MIND- and Ubiquitous Computing, Vol. 6 No. 1, Electronic
WARPING: Towards Creating a Compelling Edition (Springer LINK), 2002, pp. 75-86.
Collaborative Augmented Reality Game”, International