=Paper=
{{Paper
|id=Vol-91/paper-9
|storemode=property
|title=A Mixed Reality System for Design Engineering: Thinking, Issues and Solutions
|pdfUrl=https://ceur-ws.org/Vol-91/paperE3.pdf
|volume=Vol-91
|dblpUrl=https://dblp.org/rec/conf/mixer/NellyJ04
}}
==A Mixed Reality System for Design Engineering: Thinking, Issues and Solutions==
https://ceur-ws.org/Vol-91/paperE3.pdf
A Mixed Reality System for Design Engineering:
Thinking, Issues and Solutions
Nelly de Bonnefoy Jean-Pierre JESSEL
EADS Corporate Research Centre Toulouse Computer Science Research Institute
Toulouse Computer Science Research Institute (I.R.I.T.)
(I.R.I.T.) Paul Sabatier University
31700 Blagnac, France 31400 Toulouse, France
33(0)5 61 58 48 51 33(0)5.61.55.63.11
Nelly.de-bonnefoy@eads.net jessel@irit.fr
ABSTRACT exchanges around the physical object aims at taking some
Despite the extended use of Digital Mock-Up (DMU) decisions concerning the future product configurations such
applications during the development of complex as systems segregation, ergonomics, and physical
aeronautical products, some major changes can be arrangement of components. Consequently, during those
requested in order to correct deficiencies or to improve reviews, debates between the different designers can lead to
further products performances, especially when the first one a request for an engineering change of the product.
comes out production. These changes are discussed upon Usually, information exchanges are made verbally helped
the current product configuration and imply trades-off with hand made sketches. In those cases, it might be
between different competencies involved in the product’s difficult for a designer to make his engineering change
lifecycle. Based on a specific use case, we first highlight the proposal understood by others team members. This
underlying issues associated to the development and situation highlights a requirement implied by this kind of
introduction of Mixed Reality (MR) systems. We tackle meeting: the need to visualise the proposed modification, in
these issues from cognitive and ergonomic points of view as order to assess the potential changes impacts.
well as technical. Then, we propose several MR solutions,
which will be studied further during the two remaining Mixed Reality (MR) systems can efficiency support inter-
years of our research project. Theses solutions will then be participants information sharing around a physical object.
evaluated against the specific needs and requirements of the As matter of fact, MR systems can bring some more
aeronautic industry. engineering information about the physical object through
“optical see through”-based visualization applications. Part
Keywords changes are made visible by adding complementary
Mixed Reality, Interactions, Design Review. information to the current part («Augmented reality»
INTRODUCTION concepts)(Cf. figure1). This “augmentation of reality” can
When the first product comes out production during the be realised with the image incrustation in the users field of
development of complex aeronautical products, to correct view. Incrustation are performed through, optical see-
deficiencies or to improve further products performances, through head mounted displays. Those display devices
some major changes can be requested. They are discussed allow users to keep their environmental perception while
upon the current product configuration. These changes also having intuitively access to more contextual information.
imply trades-off between different competencies involved Therefore, such systems leverage the typical limitations of
in the product’s lifecycle. It is typically a collaborative paper-based systems, and add value to exchanges and trade-
work situation where a team of designers gather around a off through of their intuitive and fast access mode to
table to perform a product design review. Information pertinent information.
sharing and negotiation movements during those review
sessions are strongly influenced by the respective speciality,
knowledge and experience of participants. Information
Figure 1 - The Context
SYSTEM DESCRIPTION · the final change.
The investigated system allows users to virtually simulate
Depending on the nature of the information to be
the integration of an engineering change by modifying
communicated to the user, relevant output interaction
designers’ perception of the real object.
modalities have been identified. The next section present
During the meeting, all participants share the visualisation the various output modalities used and for each, the
of the actions performed on the augmented model, but the different types of information associated to them, and the
access to the model itself must remain. So that conflicts can best user-centred utilization.
be avoided. Practically, a designer must have the knack if
he desires to create virtual modifications. One design
sequence could be performed like this: he selects a virtual Output modalities
geometric form in one menu, it displays in front of him, and Three different output modalities have been considered in
he can modify its attributes by using the sub-menu order to make more perceptive actions:
associated to the virtual form. He is free to place the virtual · the visualisation through a Head Mounted Display
modification of the real object when and where he wants. (HMD),
The introduction of such systems within the review · the text-to-speech system,
environment must preserve natural communications and
social cues between participants. Indeed, the review process · and the haptic force feedback.
and user behaviour must not be perturbed by a complex MR Display of the HMD
system and its multiple accessories. The use of such system The optical see-through HMD is a display device allowing
has to be as natural and intuitive as possible, this approach users to keep their environment perception while visualising
is similar to the “Natural User Interface” (NUI) developed more information in their field of view, in context.
by [1].
Hardware
This system comes also under the definition of
The figure 2 describes the basic principle of the optical see-
Collaborative Mixed Reality systems. Some references
through Head Mounted Display:
already exist in the literature:
· “Build-it”: a collaborative mixed reality system
which tend to be a NUI [2],
· “Arthur”: the development of an interactive task
oriented collaboration environment based on
augmented reality [3],
· “MARE”: a multi-users augmented reality
environment on table set-up [4].
Figure 2 - Optical see-through HMD
INFORMATION SHARING
Information diversity There are two main devices interesting, the Sony Glasstron
Different types of information exchanges take place during (no more on the market) and the Nomad from Microvision.
review meetings, the next section focuses on system/user Equipped with this device, each designer visualizes all
information transactions. information in his field of vision, such as his environmental
We considered two major typologies of information to be information and system information. The system must
manipulated by the user: textual information, and graphical perform display in the most user-friendly and ergonomic
information. Both type can be broken down as follows: way so the user cannot be lost within the information
display density.
Text information of consists of:
For that, we discuss the general exploitation of the field of
· interface information (menus, etc.), view and we detail requirements display for each kind of
· system messages, information.
· annotations the user wants to add. Usually, people fields of view consist of one main area that
Graphical Information consists of: is a zone in the sight direction where people see all things
clearly, and around it, the field of view area where people
· arrows, chips, or others in order to show have to move eyes to see distinctly information (Cf. Figure
something to advantage, 3):
· classic geometric shape the user will manipulate in
order to obtain the final modification,
Field of view Interface information are in this area because users do not
have to focus their attention upon it. Users do not need to
keep this type of information at time in the centre of their
field of view. This area is visible only if the user has the
knack.
Direct sight access to information
System message
System messages (2) are important for users to navigate
within the application. Users must perceive this kind of
Indirect sight access to information
information as quick as possible. That is why they will be
displayed in the centre of their field of view. A special
colour has to be selected, in order to increase messages
Figure 3 - Field of view organisation
ergonomic and cognitive values.
One can be noticed that, most of the time, when people Annotation and geometric shape
have to look at something on the right (or left), even if it is
close to them, the head moves more than eyes. People tend At the beginning of the process, annotation (3) and
to put the subject of their attention in the centre of their geometric shapes (5) do not need, necessarily, to be
field of view. connected with the real object. The system will display this
kind of information in front of the user field of view, in the
Each type of information does not require the same display “direct sight access to information” area. They could be
modes. Displays do not have the same duration, neither the modified, moved, deleted in the users own way.
same location imperatives (Cf. Table 1).
Final change and Arrows, chips, etc.
LOCATION
The information category that concerned final change (6)
Specific area, Specific area, No specific and arrows, chips, etc (4), is used to emphases particular
Link with no link with area, no link items. For example, it will be used to precise a place
part part with part through an arrow or to show the real part modification
Interface (1) X through a virtual change.
Syst messag. (2) X Their location in the users’ environment is important. They
depend on the real part location in the meeting room and
Annotation (3) X X not on one specific user field of view area. The
Arrows… (4) X modification part, which is geometric shape, has an exact
location on the real object; however, arrows and others only
Geom. Shape (5) X X require an accurate location associated to the real object.
Final change (6) X General
Different colours will be used: each designer will have his
Table 1 - Information type location own colour, in order to differentiate users’ actions.
These information displays are developed hereafter. Different colours are associated to system information, one
Interface information for menus and one for messages. The selected menu section
will have different background than others, and if the user
In order to get designers used to the location of interface
goes down in sub-menus the navigation path will be
information (1), specific zones are associated to this type of
displayed as a key words list.
information (Cf. Figure 4). It is located as follows:
Field of view Moreover, users have the opportunity to remove display; all
visual information will be hidden if he desires so.
Menu
Text to speech
Sub This output modality is the sense of hearing one.
-
Menu The hardware is a headphone.
The most important part, with this modality is to have a
good thought concerning its use. If text-to-speech is used
permanently, for all information in text form, it becomes
unbearable for users very fast. And in that case, users put
Figure 4 – Interface, menu and information location the sound off.
This modality has to be used only if it gives an added value This choice has been made because interactions with the
to the system situation. That is why it is generally fitted for system must remain as intuitive as possible.
classic system messages: system announcement or error As we already mentioned there are different types of
messages. Users’ attention has to be focused on this type of information to be manipulated. The user has to interact with
information. all of them.
Haptic force feedback Speech recognition
This output modality is used to modify, and move the In this system, the integration of speech recognition is done
geometric shape that will become the final change and also in two stages.
to touch the final change. As matter of fact, the haptic force
feedback is very important to manipulate virtual objects. To First, speech recognition is used to browse menus and sub-
make the system more realistic, users must perceive tactile menus. In this case, the system must be able to recognize
information. But today, without special devices, it is almost words rather than sentences. Commercial off-the-shelf
impossible for users to catch this information from virtual applications are enough efficient to perform these functions.
objects. But a great deal of attention must be paid to the design of
menus and to the selection of a clear and concise
Hardware vocabulary.
There are two main different accessories: the glove and the In the second integration stage, the speech recognition will
Phantom, which is a computer device most closely related be used to navigate in menus, to modify virtual changes and
to the mouse. Their function is to interact with objects in a integrate them on the real product. This will imply that the
three dimensional environment. During the last few years, system will perform sentence recognition making the
many research have been carried out in this domain, and system more friendly and intuitive.
accessories progresses in ergonomics are significant.
There is, for example, the Cyberglove
(Cf. Figure5) with a vibrotactile Gestures recognition
feedback [5]. There are small Like speeches, gestures are a spontaneous mean for people
vibrotactile stimulators on each finger to communicate with other actors. The use of gestures in
and the palm of the cyberglove. They multimodal applications facilitates users interactions, in
can produce complex tactile feedback particular in noisy environments. Moreover, users tend to
patterns. Even if this glove is an execute gestures for manipulation operations rather than
accessory, the ergonomic material Figure 5 – state them or access to them by using classical interfaces
aspect is studied: users cannot feel that Cyberglove [6] like window, icon, etc.
they have a robot hand, they just wear The system use gestures recognition to interact with
an ordinary glove. interface information and to modify the shape of the
Because the system has to be as intuitive as possible and changed geometric form. This feature, in some cases, will
because there is no method without accessories, ergonomics be used simultaneously with speech recognition features.
issues play a major role. The devices that will be selected The goal is to identify and track the gestures of the user that
must allow users to keep their usual meeting behaviours. has the knack. As people know, there are different ways to
For example, they should catch objects if they want, such as make gestures recognition.
a pen, a glass, etc. Method with Digital Gloves
Flexion angles measurements, which are obtained with an
optic fibre positioned on each finger, give fingers
In an information exchange there are two communication
configuration and position. These angles are determined by
ways, in our case: “system (S) to user (U)” and “user to
the luminous signal intensity sent in the fibre and with its
system”. The last section explains how the user perceives
intensity in tip of finger. A tracker is located on the hand in
information from the system (SàU). The next section deals
order to process the hand position and orientation. This
with the User to System communication (U à S), which are
method gives accurate results, but it constraints users to
kinds of interaction the user can perform.
wear a glove generally linked to a system (depending on
technologies employed). Users do not have hands free.
INTERACTION – INPUT MODALITIES Visual Methods [6] [7]
Human-system interactions have the objective of These methods are based on computer vision and on image
developing models, concepts, tools and methods, in order to processing techniques. Hands movements are recorded with
realise systems that answer users’ needs and aptitude. one or more video cameras. Then different techniques can
Reproducing usual human-human communication modes, be used to process images depending on the gestures
the modalities used in this system are: voice and gestures. recognition method used. It is more difficult to use this kind
of method but users get rid of physical accessories. Most of As people know, there are different sorts of trackers:
the processing techniques consist of four operations, which · Electromagnetic trackers (alternating current,
are acquisition, segmentation, characteristics extraction and direct current, compass)
classification. They can be realised in different ways: based
on markers, on three-dimension model, on visual · Acoustic trackers (distance measurements
appearance. The main advantage of these methods is that determined by ultrasonic time of flight),
users do not have to wear physical artefacts. · Optical trackers (with punctual receptors
This input modality will be use to navigate in interface (phototransistors), or video based tracking),
menu and to modify and move virtual geometric shape. · Mechanical trackers (inclinometer, gyroscope,
accelerometer…),
Now that all modalities have been defined, the tracking · GPS trackers.
system, which has a primordial role in Mixed Reality The good configuration of trackers has to be found. But
System, must be tackled. some considerations have to be taken into account:
In order to make our system as natural as possible, the use
TRACKING AND REGISTRATION of peripheral is limited and bulky peripheral are proscribed.
To perform an incrustation of a virtual object in the user The less the system uses devices, the better it is.
field of view, scene components need to be located For instance, as one device used for the haptic force
accurately. Indeed, to make a good registration, MR feedback, could be also used for gestures recognition as
systems need trackers with approximately one millimetre well.
accuracy in position and a low fraction of degree in
Moreover, this use case takes place in a specific meeting
orientation. Most of commercially available trackers answer
room that is a well-defined place and where the luminosity
one of the two conditions but not both.
is constant.
Tracked elements
Considering the huge advances made in video technologies
In order to offer an intuitive and a free visualisation of part during the last few years, it is now possible to find little
modification, the system will track continuously and with cameras with very good resolution. This greatly improves
accuracy different environment elements: the quality of image processing for markers recognition.
· the physical part, Moreover these video cameras are now equipped with USB
· each designer, communication port, providing a good data quality and
speed transfer. Practically all web cams have these
· users points of view, characteristics today, so a good camera could be found for a
· users hands. reasonable price. To finish with, little cameras start to be
equipped of IEEE communication port, which offers the
The first three elements are tracked to make an efficient
best transfer speed and quality.
registration of the virtual modification on the real object in
each user field of vision. The last one is tracked so users
can realize virtual changes with gesture recognition, move For all these, the video based tracking has been selected for
the virtual object, and perform haptic force feedback. our system. Possible video based tracking methods are
The tracking system is designed to create relationships presented in the next section.
between each tracked element configuration (Cf. figure 6).
Video based tracking methods [8]
As people know there are two main video-tracking
configurations:
· Inside out (Cf. Figure 7)
One or many video cameras are on the moving target. They
watch markers fixed in the environment which are
references.
· Outside in (Cf. Figure 8)
One or many video cameras are fixed in the environment,
they are references; they watch the target movements, on
Figure 6 - Referential example
which markers have been affixed.
Glasstron and the Nomad of Microvision. Their display
length on the field of vision can be considered as small.
Devices Resolution Display length
Sony Glasstron 800 x 225
Microvision HMD 800 x 600 23° x 17°
There are other display devices on market, based on the
head mounted display concept, like video-see through
Figure 7 - Inside-out Figure 8 - Outside-in
HMD and some video screens that can be clipped on
Markers should be classic draws or LEDs. Once the glasses. However, the first one does not allow users to see
configuration is chosen, there are different manners to their close environment directly, and the second type
calculate the target location. The first uses the two or more proposes a small display of a computer screen, PDA screen,
cameras in order to calculate the target marker positions, by etc.
using the triangulation for example. The target orientation
should be calculated by using several markers on it. The
second use the pattern recognition techniques; there is only Real object location issue
one video camera and some target markers geometric The real part is on a Location of the First extension
2nd extension
knowledge. table. There is a
visualisation problem
The fact that the inside-out method gives more accurate
(Cf. figure 9) if
results and a better orientation resolution than the outside-in
different changes are
method should be noticed.
made in different
locations of the real
Registration part (one of the bottom Figure 9 - problem visualisation
The registration is one recurrent problem in such system. In and another at the top
order to make the visualisation of the real object and its for instance).
virtual change realistic, an accurate registration is required. The real object location needs a thorough thought in order
Two types of errors can be encountered: the static and the to permit users the visualisation of different changes.
dynamic ones [9].
The static errors are due to the optical distortion, errors of Gestures recognition issues
the tracking system and differences between models or Gestures recognition using visual approach, in particular
material specifications and real material physical properties. without the use of markers, is not a mature technology.
This kind of errors is perceived even if the user does not Some systems work in real time but in very specific
move. conditions, such as with a uniform background, a small
The dynamic errors are due to the processing time lag that vocabulary, etc. Even if a system that recognises gestures,
is the delay between measurements made by the tracking with a visual approach without markers, in real time, in a
system and the display of the virtual entity. In fact, there are normal environment would add a tremendous value to
due to all processing time devices and systems. Different general MR systems, it remains, for the moment, a research
ways have been explored to reduce this dynamic error, by perspective.
reducing the system lag or the perceptible delay [11], by Moreover, as people know, in the human-human
making location prediction [11], by image matching. communication, the use of speech, gestures, and facial
To make a good registration the tracking system must expressions contributes to the information exchange. In
perform good locations. particular during argumentative phases or solution
negotiation activities, the frequency of gestures and facial
To resolve the static errors a calibration have to be made,
expressions significantly increases. The case study
and as many system, ours will use Kalman filtering to
presented in this paper deals with one specific situation
reduce dynamic errors.
where people will try to negotiate integrating several
different points of view. So, making the difference between
SYSTEM ISSUES social communication gestures with actual gestures
performed to interact with the system is primordial in our
Hardware issues: Optical see-through Head Mounted
Display case. The system must have the possibility to distinguish
For the moment this technology is not mature enough. As these two types of gesture, in order to avoid false
we saw upper, there are two main devices, the Sony interpretations.
Virtual modifications tracking system and issues met. All these bases will allow
Object that people can bring in a meeting room are us to build a prototype.
generally medium-sized. Moreover, changes made on a part
during the review sessions are not consistent; the general
object shape is not called into question as modifications are REFERENCES
localised and concern only small areas. Therefore, to keep 1. Rauterberg, M., From Gesture to Action: Natural User
the visualisation as realistic as possible, the registration of Interfaces. Eindhoven University of technology; Mens-
the virtual change has to be very accurate. Machine Interactie, Diesrede 1999, 15-25.
Part modification is not necessary an extension of the real 2. Fjeld, M., Bichsel, M. and Rauterberg, M.: BUILD-IT:
object, it could be a suppression of a little part area. To Intuitive plan layout mediated by natural interaction.
make an extension the system proposes to the user to mould Arbete Människa Miljö & Nordisk Ergonomi (work,
a classic geometric form and after to register it on the real human being, environment), 1/99, 49-56.
object. But to make small area suppression, it is more 3. Romell, O., Arthur (augmented Round Table for
difficult to make it realistic, the representation is more Architecture and Urban Planning) – An augmented
elaborated. Reality, Task Oriented Collaboration Environment. AVR
The system has the three-dimension model of the real II and CONVR 2001 Conferences at Chalmers,
object. This will help us to make this kind of modification Gothenburg, Sweden, October 4-5, 2001, 29-37.
as realistic as possible. 4. Grasset., R., and Gascuel, J.D.: MARE: Multiuser
Our first solution is to use a dark shape to simulate the Augmented Reality Environment on table set-up.
suppression area. But the realistic visualisation depends too 5. http://www.immersion.com/3d/products/cyber_touch.
much on the user’s location (Cf. Figure10).
6. http://www.immersion.com/
Dark shape
7. Ying Wu, Thomas S. Huang, "Vision-Based Gesture
Recognition: A Review", A. Braffort et al. (Ed.) Lecture
Notes in Artificial Intelligence 1739, Gesture-Based
Communication in Human-Computer Interaction
(International Gesture Workshop, GW'99, Gif-sur-
Yvette, France, March 1999).
8. Ying Wu, "Vision and Learning for Intelligent Human-
Figure 10 - different modification views Computer Interaction", Ph.D. Dissertation, 2001,
Department of Electrical and Computer Engineering,
University of Illinois at Urbana-Champaign.
Occlusions issues
9. Rolland, P.J., Baillot, Y., and Goon A., A survey of
Occlusion is the typical issue of this type of MR system.
tracking technology for virtual environments. Center for
All actors are situated in a close room area. In our study research and education in optics lasers (CREOL),
case, the design review is made around a table; so all users university of central Florida, Orlando FL 32816, 1999.
move and perform their task, in a confined place. In this
10. Azuma, R., A survey of augmented reality. In presence:
context, occlusions will be frequent.
teleoperators and virtual environment 6, 4, August
Solutions must be found in the room layout, in cameras 1997, pp.355-385
positioning and in tracking methods.
11. Kijima, R., and Ojika, T., Reflex HMD to Compensate
Lag and Correction of Derivative Deformation, IEEE
CONCLUSION Virtual Reality Conference, Orlando, Florida, March 24
This paper deals with information sharing in our MR - 28, 2002, pp.172-182.
system for design engineering, the modalities used, the