=Paper=
{{Paper
|id=Vol-2326/paper6
|storemode=property
|title=MVC-3D : Adaptive Design Pattern for Virtual and Augmented Reality Systems
|pdfUrl=https://ceur-ws.org/Vol-2326/paper6.pdf
|volume=Vol-2326
|authors=Samir Benbelkacem,Djamel Aouam,Nadia Zenati-Henda,Abdelkader Bellarbi,Ahmed Bouhana,Samir Otmane
|dblpUrl=https://dblp.org/rec/conf/icaase/BenbelkacemAZBB18
}}
==MVC-3D : Adaptive Design Pattern for Virtual and Augmented Reality Systems==
https://ceur-ws.org/Vol-2326/paper6.pdf
MVC-3D: Adaptive Design Pattern for Virtual and Augmented Reality
Systems
Samir Benbelkacem, Djamel Abdelkader Bellarbi,Ahmed Samir Otmane
Aouam, Nadia Zenati-Henda Bouhena Laboratoire IBISC, University of
Centre de Développement des Centre de Développement des Evry.
technologies Avancées. technologies Avancées. 40 rue du Pelvoux, Evry, Paris,
Cité 20 août 1956, Baba-Hassen, Cité 20 août 1956, Baba-Hassen, France
Algiers Algiers samir.otmane@ibisc.univ-evry.fr
sbenbelkacem@cdta.dz
researches focus on technical issues such as, tracking,
rendering, I/O devices in order to design such interfaces. The
lack of design models and interdisciplinary nature of virtual
Abstract and augmented (or mixed) reality design require design
approaches based on iterative refinement. Also, the lack of
In this paper, we present MVC-3D design pattern formal verification techniques and the limited design
to develop virtual and augmented (or mixed) reality expertise limits the applicability of techniques like experts
interfaces that use new types of sensors, modalities reviews for NGUI applications, except for experimental
and implement specific algorithms and simulation evaluation through tests with end-users that could be a
models. The proposed pattern represents the potential solution. However, a simple implement and test
extension of classic MVC pattern by enriching the approach is not viable because the implementation of
View component (interactive View) and adding a working prototype is expensive and time consuming, limiting
specific component (Library). The results obtained the number of concepts and designs that can be explored. To
on the development of augmented reality interfaces overcome these problems, we propose an adaptive design
showed that the complexity of M, iV and C pattern, with supplementary concepts specific to virtual and
components is reduced. The complexity increases augmented reality systems. This paper briefly illustrates the
principal idea of our design approach in section 3 and
only on the Library component (L). This helps the
presents preliminary results in section 4. But we start with a
programmers to well structure their models even if
short review of related work in section 2.
the interface complexity increases. The proposed
design pattern is also used in a design process
called MVC-3D in the loop that enables a seamless 2. Related work
evolution from initial prototype to the final system.
Design models have been proposed for the
Keywords - Virtual and Augmented reality, Design development of virtual and augmented reality interfaces
Pattern, Design Process in the literature [Emm 17]. Ishii [Ish 08] extended the
MVC model [Bur 92] to develop tangible interfaces.
1. Introduction Stöcklein [Sto 09] proposed MVCE model that
integrates “Environment” component into the MVC
The evolution of the interaction between a user and model. In [Tar 97], AMF-C design pattern was
mechatronic systems require a new design of task-adequate proposed to design mixed reality collaborative systems.
and usable user interfaces. This leads to a new generation of
In our previous works [Ben 14], we realized three
user interfaces (NGUIs) that provide a set of new
technologies of human-computer interaction based on newly
implementations of a fragment of the same application
developed modalities, sensors and algorithms. NGUIs “car engine maintenance assistance using Augmented
diverge from the well-known WIMP (Window Icon, Menu Reality”. The first implementation used a traditional
and Pointer) and use novel interaction paradigms such as MVC model [Bur 92]. The second implementation used
Virtual Reality (VR) and Augmented Reality (AR), tangible, an MVC pattern implemented using a communication
embodied and multi-modal interfaces. In recent years, pattern proposed by Eckstein [Eck 07]. The third
Copyright © by the paper’s authors. Copying permitted only for private
implementation was based on an AMF-C architecture
and academic purposes. model [Tar 97]. The programmers estimated that
structuring the code into components can be complex
In: Proceedings of the 3rd Edition of the International Conference on
Advanced Aspects of Software Engineering (ICAASE18), Constantine, using the MVC model. The complexity of components
Algeria, 1,2-December-2018, published at http://ceur-ws.org
Page 51
MVC-3D: Adaptive Design Pattern for Virtual and Augmented Reality Systems ICAASE'2018
increases when implementing AMF-C model. In the additional components and features that consider
other hand, we observed that the MVCE model [Sto independently all processing specific to VR & AR (e.g.
09] increases the complexity of components when the tracking algorithms) and integrate the environment
interface contains more 3D information and integrates (virtual and real) with their corresponding interactive
many processing (e.g. simulation tools, tracking and devices. For tangible interfaces we are based on the
gesture recognition algorithms). In addition, concepts presented in [Ish 08].
introducing complex simulation models and heavy The benefit of the classical MVC pattern is the
calculations in Model (M) and Environment (E) separation of interaction and visualization aspects of
components makes the development tedious and upsets the user interface when designing the application. Also,
programmers’ practices. it enables modular designs with which changes
In our approach, we integrate an additional performed on a component doesn’t affect other
component to MVC model that consider all tools, SDK components. The Model (M) encapsulates the
and heavy processing separately. When a technology is application data and the functionalities of the
modified (simulation toolkits, algorithms and devices), application. The Controller (C) handles the user actions
we change only the additional component’s content and plays an intermediate role between View and
without affecting the Controller and the Model. Only Model. The View (V) represents the visual items of the
adaptation can be made for these two components to application. One of the key elements of virtual and
support the changes. With this approach, we can reduce augmented (or mixed) reality user interfaces is the
the complexity and recurrent accesses to MVC maintaining of coherent relationship between real world
components and preserve the programmers’ practices. and virtual models in real time when incorporating
heterogeneous software and hardware technologies. To
3. Design Approach guarantee this coherence, the structure of the
components should be less complex as possible to
The goal of many augmented and virtual-reality facilitate the information exchange and reduce
applications is the development of better user interfaces processing time. Our vision is to integrate heavy
based on an iterative design process. However, most processing of the application in a specific component
design processes assume that the underlying technology which could be modular and reusable. Therefore, we
is well defined and stable, a condition that is often not introduced an additional component “Library” to MVC
respected in the development of NGUIs which use model (see Fig.1) that encapsulates all processing using
developing technologies that are still in early specific algorithms (e.g. tracking techniques, gestures,
experimental stages. Changes in the underlying faces and speech recognition algorithms), complex
technologies can be problematic, if a design process simulation models (e.g. Matlab/Simulink) and
does not anticipate such volatility and therefore SDK/toolkits to process the amount of data provided by
provides no means to handle it in a systematic and the View via the Controller.
structured way. Also, most of the virtual and In addition, some interaction devices should be in
augmented reality prototypes are designed without any the user's view field to perform 3D manipulation tasks
structuration and using a Trial-and-Error prototyping (gestures interaction, speech control…). From practical
approach. This leads to an overhead in the point of view, we observed that it is difficult to
programming, since parts of the application cannot be dissociate physically both the environment and
reused and must be re-implemented for each prototype. interactive devices from the view of the user. For that
In our approach, we have developed an iterative design purpose, we brainstormed to conceptually merge these
process which can be used for NGUIs based on our elements. We proposed to enhance the View
proposed MVC improved model. component by introducing a sub-component that
captures the real environment model of the application
3.1 MVC-3D Design Pattern and a sub-component that integrates the sensors module
and manages the tangible objects (physical and digital).
We present an adaptive design pattern to implement We obtained then a new component that we call
interaction techniques of virtual and augmented reality interactive View (iV) (see Fig.1).
interfaces. We propose a structure that extends the Using the MVC-3D structure, components can be
(MVC) model-view-controller design pattern with refined independently.
International Conference on Advanced Aspects of Software Engineering Page 52
ICAASE, December, 01-02, 2018
MVC-3D: Adaptive Design Pattern for Virtual and Augmented Reality Systems ICAASE'2018
from these entities, connecting the information flow
interactive View (iV) Library (L) between the input and output ports as required by the
User actions application. For the technical implementation we use
Notification
data Update and data
data our mixed reality environment described in [Ben 14].
Query data and display Send data New data
Specification and
Query data/update data analysis phases
Model (M) Controller (C)
Send data
Figure1: MVC-3D Model
Add/remove/split/
One key benefit is the possibility to develop an
interactive user interface along the mixed reality update entities
continuum [Mil 94] in which dedicated processing is
formulated by a component: Library (L). This Previous Next
component involves heterogenous and complexe Prototype Prototype
simulation models, SDKs and algorithms. The benefit
of this component is the capability to manage this
Refinement of
heterogeneity and the relationship between the different MVC-3D model
models. If we plan to change an algorithm or a toolkit,
we just modify the Library content without completely
changing the Model and Controller's content. A simple
adaptation could be made.
Coding
3.2 Design Process
Figure 2: MVC-3D in the loop
Our goal is to iteratively develop virtual and augmented
reality prototypes that could be refined to correspond Typically, the first prototype consists of an
the user’s expectations. These prototypes can be then environment in which all entities are implemented
used to evaluate the system with end-users and to purely in software.
validate the technologies used in the system. Using the The resulting system would therefore be a purely
MVC-3D model, components can be refined virtual environment. The key benefit is that elements in
independently. The benefit of this approach is the a virtual environment are faster and cheaper to develop.
possibility to develop a user interface along the mixed Depending on the development goals and priorities,
reality continuum [Mil 94] starting from purely virtual entities are selected for refinement.
environment to real one. Figure 2 describes our design Refinement means that either the behavior or visual
process. representation of an entity is updated (e.g. the 3D-
We define each MVC-3D component as an ’entity’. model) or an entity replaced by another version of the
An entity can be anything from a model, a visual same entity (e.g. Game physics vs. Matlab/Simulink). If
representation, to a controller. the entity is concerned with simulating real-world
At the beginning of the development process an elements, typical refinements would be the replacement
initial set of entities is identified. For each entity in this of a simple simulation with a more realistic one, or the
set the inputs and outputs are defined. If the replacement of a simulation component with its real
development targets a complex mechatronic system, world equal. This approach allows to move from purely
each relevant system component (either hardware or virtual environments to hardware in the loop (or more
software) is initially represented by an entity. precisely mixed reality in the loop) systems in a
Additional entities represent the elements of the user structured way. If an entity is concerned with the
interface. This initial set of entities can later be implementation of interaction or visualization
extended. It should include all entities needed for a first techniques the replacement could either be a complete
prototype that provides a rough approximation of the exchange of the component (e.g. to compare alternative
planned system. The first prototype is then composed approaches to system control) or stepwise refinements,
International Conference on Advanced Aspects of Software Engineering Page 53
ICAASE, December, 01-02, 2018
MVC-3D: Adaptive Design Pattern for Virtual and Augmented Reality Systems ICAASE'2018
in which a user interface is refined according to user algorithms calculating the position and orientation of a
feedback as in established iterative user centered design hand and its tracking in the AR scene. The processing
processes. As the development progresses from an linked to the gesture recognition was not integrated into
initial basic prototype to more complex systems, it can the Controller (C) and the classical View (V), neither in
also be necessary to adjust the number of entities, e.g. the Model (M), but in the Library (L). The Model (M)
by splitting the functionality of an entity into two or is enriched by adding the gestures’ data. The Controller
more, or by adding or removing entities. In all cases the (C) manages both the images and the gestures data
data flow connections between the entities must be which are provided by the Library (L). The interactive
checked and corrected consequently. View (iV) encapsulates the Kinect module and the
user’s hands data besides of the real cylinder head’s
4. Preliminary Results reference image data. Therefore, interactive View (iV)
component facilitates data access (e.g. the Kinect
We have applied the MVC-3D model during the delivers directly gestures’ data to the classical View
implementation phase of the extended 2TUP process (V)). The Controller (C) can easily handle data arising
[Ben 14] for the fragment of the application from the real environment.
"maintenance of a car engine by AR". We have In the same manner of the precedent case, prototype
developed three prototypes with different interaction 3 is the refinement of prototype 2. We have added to
techniques using MVC-3D pattern for each one. prototype 2 two entities: (1) a Kinect with 3Gear
Preliminary results are given in Table 1. library that capture the gesture motion and (2) the AR -
Prototype 1 shows an AR interface of a real cylinder Head Mounted Display (HMD) to view the 3D piston
head of a car engine on which are displayed four 3D manipulated by the user. So, the MVC-3D is refined
pistons (interactive View (iV)). The Model (M) along Model (M), interactive View (iV), Controller (C)
encapsulates 3D pistons’ Data, the transformation and Library (L) axes (see row 3 of Table 1).
matrix of the 3D pistons and the markers’ Data. The Comparing prototypes 2 and 3 using our approach,
Controller (C) manages the images’ data provided by the structure and content of the Model and Controller
the camera, sends the data to the Library (L) and have not deeply changed; a simple adaptation and/or
receives the results. The Library (L) uses ARtoolKit enhancement have been made. Complex algorithms and
tracking algorithm to process the data provided by the models are integrated into the Library instead of Model
Controller (C). and Controller comparing to the approaches presented
Prototype 2 displays the same augmented scene, but in the literature where all the components increases in
without markers (interactive View (iV)). The Library complexity. In the other hand, the structure of the
(L) implement MOBIL algorithm [Bel 14] for marker- interactive View can be relevant since it promotes
less tracking system. The programmers add to the exchanging data between the classical View, the
Model (M), already established for prototype 1, the environment model and interactive devices.
reference image' data of the cylinder head. The
Controller (C) was adjusted to support exchanging data 5. Conclusion
with the Library supporting MOBIL algorithm. In this
case, the content of the Library was modified by To consider virtual and augmented reality systems’
implementing MOBIL algorithm. Therefore, the specifications in a design process, a design approach is
complexity increases only at the Library (L) component required. In this paper we presented the “virtual and
in which processing to detect and track the scene is augmented reality in the loop” process, based on
more consistent. Prototype 2 is the refinement of structuring the VR & AR applications into Model,
prototype 1 using the design process given in Fig.2. In interactive View, Controller and Library components
fact, we have removed an entity (ARtoolKit SDK) and that can be refined individually. We have shown how
replaced it by a new entity (MOBIL Marker-less this model was used successfully to refine prototypes.
technique). So, the MVC-3D is refined along Library In the other hand, we detailed MVC-3D design model.
(L) axis (see row 2, column 4 of Table 1). This model can help programmers to better structure
Prototype 3 is an extension of prototype 2. It the programming process and focuses specific
provides an interactive interface where the user processing in the Library component with which the
manipulates the 3D pistons using his hands. The implementation of methods and tools can be well
Library encapsulates, besides MOBIL tracking system, organized.
International Conference on Advanced Aspects of Software Engineering Page 54
ICAASE, December, 01-02, 2018
MVC-3D: Adaptive Design Pattern for Virtual and Augmented Reality Systems ICAASE'2018
Table1: Implementation of three prototypes using MVC-3D model
Model (M) Controller (C) interactive View (iV) Library (L) Complexity measure Prototype
- Transformation - Processing image - Four 3D pistons. - Recognition of M iV
matrix of the 3D provided by camera: simple square
Pistons. convert image (pre- - Environment: markers using
processing) and cylinder head and ARtoolkit.
- 3D pistons’ Data inserts 3D pistons square markers.
into image (post-
Complexity
- Markers’ Data. processing). - Sensors module:
camera.
C L
- Transformation - Processing image - Four 3D pistons. - Detection and
M iV
matrix of the 3D provided by camera: recognition of
Pistons. convert image (pre- - Environment: reference image
processing) and cylinder head. (cylinder head’s
- 3D pistons’ Data. inserts 3D pistons image) using
into image (post- - Sensors module: MOBIL algorithm. Complexity
- Reference image processing). camera.
data (Cylinder head’s - Camera pose
image). estimation.
C L
- Transformation - Processing image - Four 3D pistons. - Detection and
matrix of the 3D provided by camera: recognition of M iV
Pistons. convert image (pre- - Environment: reference image
processing) and cylinder head. (cylinder head’s
- 3D pistons’ Data. inserts 3D pistons image) using
into image (post- - Sensors module: MOBIL algorithm.
- Reference image processing). camera and Kinect. Complexity
data (Cylinder head’s - Camera pose
image). - Gesture based estimation.
control.
- Gestures’ Data. - Gesture C L
recognition.
MVC-3D can be also adapted for applications [Emm 17] F. Emmerich, R. Klemke, T. Hummes.
involving heterogeneous algorithms and different Design Patterns for Augmented Reality
interaction devises. Using our design approach, we can Learning Games. In: Dias J., Santos P.,
preserve the programmers’ practices and reduce Veltkamp R. (eds) Games and Learning
programming complexity. Alliance. GALA 2017. Lecture Notes in
Computer Science, vol 10653. Springer,
References Cham.
[Ish 08] H. Ishii, “The tangible user interface and its
[Bel 14] A. Bellarbi, S. Otmane, N. Zenati-Henda, S. evolution”. Communications of the ACM, Vol.
Benbelkacem, “MOBIL: A Moments based 51, No. 6, pp. 32-36, (2008).
Local Binary Descriptor”, 13th IEEE [Mil 94] P. Milgram, F. Kishino, “A taxonomy of
International Symposium on Mixed and mixed reality visual display”, IEICE
Augmented Reality (ISMAR 2014), pp.251- Transactions on Information and Systems, E77-
252, Munich, Germany, (2014). D(12), January, (1994).
[Ben 14] S. Benbelkacem, M. Belhocine, N. Zenati- [Stö 09] J. Stöcklein, C. Geiger, V. Paelke, P.
Henda, A. Bellarbi, M. Tadjine “Integrating Pogscheba, “MVCE - a design pattern to guide
human-computer interaction and business the development of next generation user
practices for mixed reality systems design: a interfaces”, IEEE Symposium on 3D User
case study”, IET Software 8(2), 86-101, (2014). Interfaces (3DUI), pp.153 – 154, Lafayette, LA,
[Bur 92] S. Burbeck, “Applications Programming in (2009).
Smalltalk-80(TM): How to use Model–View– [Tra 97] F. Tarpin-Bernard, “Travail coopératif
Controller (MVC)”, (1992). synchrone assisté par ordinateur : Approche
[Eck 07] R. Eckstein, “Java SE Application Design AMF-C”, PhD thesis, Ecole Centrale de Lyon,
With MVC”. Sun Microsystems Inc., (2007). France, (1997).
International Conference on Advanced Aspects of Software Engineering Page 55
ICAASE, December, 01-02, 2018