A 3-level modeling framework to adaptive resource management in mobile augmented reality systems (MARS) is proposed, which is based on comprehensive data structuring and analyzing of their specific hard- and software features. At the conceptual modeling level an ontological specification of MARS resources is constructed, at the logical modeling level a case-based reasoning algorithm is elaborated, and as a physical model the reference software architecture is designed. This approach was successfully tested to solve the task to adaptive management of image resolution on mobile device (MD) according to changes of computational loading that finally enabled better video stream quality in MARS.
Nowadays, mobile information systems become more and more popular. One of the
most complex and dynamically grown type of these systems are mobile augmented
reality systems (MARS) [
- 42 mation systems development, in particular for MARS. Such systems development requires effective utilization of restricted mobile device (MD) resources and on the other hand implementation of complex real-time algorithms.
In this paper we propose an adaptive model-based framework for resource management in MARS. Additionally a prototype implementation of adaptive MARS is presented and series of experiments with proposed framework and MARS are provided. These experiments highlight positive effect of using adaptive approaches in MARS development.
This paper is structured in the following way: Section 2 depicts briefly some modern trends in this research domain, with respect to some adaptation issues; Section 3 provides main concept of adaptive model-based framework for resource management in MARS; Section 4 presents some ARS domain modeling issues like ontological specifications; in Section 5 the algorithmic model for adaptive resources management and metrics are given; in Section 6 presents proof of concept, introduces software prototype and experimental results. Finally, a short outlook on the result achieved and some future work is presented. 2
An Augmented reality (AR) is a representational form for a real physical
environment, which is extended by adding of computer-generated data [
Currently AR supports several data sources: two-dimensional markers; data
received from GPS-modules (Global Positioning System) [
With respect to supported data source, it is possible to construct some MARS classification, which is presented in Fig. 1. There are four “top-level” types of MARS; each of these types has some features and requirements to MD performance. Markerbased MARS operates with special markers, which stores some data and links to additional information. Non-marker based MARS are more complex types of these systems, it based on image recognition algorithms and requires more resources to find and recognize free-form objects in an input image. Geolocation MARS uses build-in GPS sensors in order to get information about real environment and augment it with some virtual data. Finally, infrared-sensors based MARS operates with some infrared sensors, which is able to detect object and moves in real environment, subsequently this class of MARS is very useful in entertainment and simulators. In scope of this research we are focused on Markes-based MARS, because this class of MARS has the lowest complexity, easy to implement and does not require any additional equipment apart from MD.
- 43
In terms of software development there are some modern software frameworks to
develop MARS: Metaio Mobile SDK (Software Development Kit) [
Nowadays, software adaptation is one of the common trends in modern software
engineering (see, e.g., in [
Q-CAD is a resource discovery framework which enables mobile applications to
dis-cover and to select resources best satisfied the user’s needs. MADAM and
ITSMUSIC frameworks provide model-driven development approach enabling to
assemble applications through a recursive composition process. In this case, variability is
achieved by plugging into the same component type different component's
implementation with similar functional behavior [
Elastic Application Model [
To sum up related work investigation we can conclude that none of existing approached doesn’t provide complete model based framework to adaptive resource management in MARS, but take into account particular aspects of this problem.
- 44
In the Fig. 2 and in the Fig. 3, the interface of marker-based MARS is presented. This MARS detects source object in the input video stream recognize it, obtain 3D graphical model of this gear and finally augment source image with this model. In the Fig. 3 result image is shown, MARS finds marker in the source image (in this case – gear), obtains related data and augments source image with this data. In this case, result of augmentation is the gear’s 3D model which is presented over sources gear image.
Marker-based MARS [
Taking into account the results of the provided analysis and based on the
understanding of modern trends in the domain of adaptive MARS-development (see Section 2),
we can conclude that it is necessary to elaborate a complex model-based framework
for adaptive resource management in MARS. This assumption is completely
corresponded with such well-proved and recognized approaches in modern software
development as model-driven development (MDD) and model-driven architecture (MDA)
[
The interacting between these abstraction levels which build the proposed modelbased framework is shown in Fig. 4.
In this way, it is possible to elaborate a lot of knowledge-oriented and reusable algorithm-centered solutions and software components, which support the adaptive approach to resource management in MARS. More detailed these issues are considered below. 4
Taking into account the 3-level modeling framework for adaptive resource management in MARS proposed in previous Section 3, it is needed to elaborate the domain model for this purpose. This model should represent all relevant hard- and software capabilities of MD which can be considered as adaptable parameters in an appropriate algorithmic modeling approach.
Ontology models are widely used to represent relationships between concepts in
some application domain, and they can be applied for different purposes in software
engineering, e.g.:
1. for information sharing between human and machines in Semantic Web
applications [
For example, [
Most of models mentioned above describe different MARS components taking into
account some static resource allocation, and they do not represent system features
needed for adaptive resource management. To close this gap in our approach the new
MARS model was created using the OWL notation, and the OWLGrEd tool is used
for this purpose [
There are following main entities included in our domain model 1. Augmented Reality Application: is an application that provides a technology to analyze elements in the real physical environment, and to extend them with additional virtual objects or with additional information. 2. ACU (Adaptive Control Unit): this is control management element that is responsible to adapt system to the mobile device. It could use different techniques to optimize application, e.g. to change display resolution. 3. Mobile Device: this is a small device (iPhone, PDA, notebook, tablet etc.), with restricted amount of system resources, which does not have permanent access to the power sources, and which uses wireless communication technologies. For resource adaptations, the following device parts (model entities) could be used: Screen – by changing screen resolution performance of the device could be changed; RAM – depending on the allocated memory different application modes might be used; Battery - usually a charged level is used as a main parameter to switch device into power save mode; modern CPU could change frequency and switch its operation mode. 4. AR Object Recognizer: is a module to analyze and to extend extracted physical objects with additional or virtual information.
Below this domain model is used to elaborate an algorithmic approach to adaptive resource management in MARS. 5
According to the proposed adaptive model-based framework (see Section 3) at its
logical level an algorithmic model to resources management has be constructed with
respect to specific hard- and software characteristics of MD which is used to get client
applications running within a given MARS.
In order to formalize the proposed framework to find adaptive solutions for resource
management we can use an algorithmic modeling approach [
AM WorkflowMethods, InfoBase, Metrics ,
(1)
where Workflow Methods are some algorithms which implement the given methods
Methods , InfoBase is an information base to be used for these methods, and Metrics is
a collection of metrics to assess a quality of adaptation process in mobile ARS. The
choice of a set of adaptation methods in (1) depend on specific features of ARS’s
resources, which should be managed, and one of such possible way will be considered
in the next subsection. A set of metrics in (1) also has to be defined taking into
account the appropriate hard- and software properties of MD which is used in a target
ARS (see e.g. in [
Taking into account a complex and weak-formalized character of ARS – functioning,
namely: parallel and multi-threaded calculation processes, turbulence loading on MD,
permanent changes on number of users etc., it is reasonable to use so-called soft
calculation methods [
(2)
where NNM is a Nearest Neighbor Method, kNNM is a k-Nearest Neighbors Method,
and kwNNM is k-weighted Nearest Neighbors Method [
The main idea of all CBR-methods is that any new problem occurred in some application domain can be resolved using already existing solution for the similar situation (called precedent or case). The several CBR differ each other in a search algorithm to find an appropriate precedent in the given case - database. For this purpose, it is also important to elaborate an adequate description for the precedent’s representation, which reflects all relevant issues of ARS functionality. 5.3
Corresponding to formula (1), InfoBase is an information base which is used to apply the CBR-methods defined in (2). It includes a set of precedents, and any such precedent can be defined in the following way
where p is a vector of parameters to characterize a given problem situation, s is a vector of parameters to represent an appropriate solution for this problem.
Taking intoaccount the hardware - and software issues of MD which is included in ARS, vector p can be given as: p CPU , RAM, BAT, RES, FPS , where Width and Height are respectively a width and a height of a video frame on MD. where CPU is a current level of processor loading (in %), RAM is a current level of RAM usage, BAT is a current level of battery charging; RES is a number of possible screen resolution modes in MD, FPS is a measure of a screen refresh rate.
Vector s in formula (2) can be represented as the tuple s Width, Hight , 5.4
In [
Therefore, a collection of metrics Metrics in (1) has the following definition s T , R , (6) where T is a metric to estimate a number of frame per second, P is a metric to measure a MD total pRoductivity (R).
A value of metric Т can be calculated using the standard function Count(), namely: (3) (4) (5) (7) (8)
T Count FPS R wc CPU total RAM cur wb BATtotal
CPU cur wr RAM total
A value of metric R (named below as a productivity index) defines a MD performance ratio, which is dimensionless parameter and it can be defined as following where CPUcur is a current MD processor loading ratio (in %); RAMcur is a current RAM usage ratio (in Kb); BATcur is a current battery charging ratio (in Ah); CPUtotal, RAMtotal , BATtotal are respectively the nominal values of the given parameters; wc, wr, wb are some weighting coefficients for these parameters, and the following condition must be fulfilled wc wr wb 1. In this work we have defined the following value ranges for the index R: it is critical if R 0.95; it is high if 0.6 R 0.95; it is normal if 0.25 R 0.6 ; and it is low if 0 R 0.25 .
Therefore, the metrics defined in formula (6) – (8) allow us to estimate the computational resources of an appropriate MD which is used in a target ARS with respect to our final goal: to provide an adaptive recourses management in this ARS. 6
6.1
In order to prove efficiency of the proposed approach the MARS prototype with integrated ACU has been developed. The main purpose of developed MARS is to recognize marker on a cinema poster, search information about this cinema in an appropriate database and augment source video stream with this additional information at realtime mode. Such MARS with integrated ACU which analyzes environmental parameters and adopts frame size with respect to MD current state and resources utilization rate. In the Fig. 6 is presented adaptive MARS functioning algorithm in form of UML activity diagram. Initial activity in this algorithm is – calculation of a productivity index. If this index is less than 0.95 (R ≤ 0.95) it is possible to augment data on the MD side, so the next steps are to define adaptive video stream size, using CBR method, and augment source video stream with a virtual data on a MD side, and finally – display augmented video stream to user. If index R > 0.95 it is not possible to augment data on the MD side and in this case it is necessary to use external resources to augment image from input video stream (e.g. server in client-server MARS) and show result to user. This activity could be interrupted by user’s event (pressing exit button).
Presented algorithm takes into account some code offloading possibility (in case of quite high computation load on a MD), subsequently it is useful to select three-tier software architecture to implement adaptive MARS prototype. Such architecture allows us to implement recognition component on the server side in order to offload some logic from MD in case of critical computational load. Another plus point of this architecture is a possibility to deploy centralized precedent database on the server side and distribute this precedents among different MDs. In the Fig. 7 presented three-tier component software architecture of MARS in form of UML deployment diagram. This architecture provides few crucial advantages such as: high scalability, data processing security, lower resource requirements for clients MD.
Client MARS application implemented with Android platform [
ACU on a MD node (Android Device) contains the following components: Precedent-Storage – local precedent DB, SystemMonitor – the component which provides data regarding current state of MD and ACU – the component, which implements CBR methods on the mobile client side. On the server node into adaptation process are involved the following components: Precedent DB Processor is the accessor component for centralized Precedents DB, and Data Analyzer is the component that implements movie's additional data search by input images hash-code.
With respect to the proposed architectural solution, three databases have been
implemented: 1) local embedded precedents DB (PrecedentStorage), this DB is used by
ACU; 2) the remote centralized DB (Precedent DB) to store all precedents, all local
DBs are synchronized with this one; 3) movies DB (Movies DB), this DB stores
information about movies, which are handled in MARS prototype. Conceptual data
model of the Precedent DB is presented in the Fig. 8 as the UML class diagram. This
data model takes into account the following entities: Device, Precedent, Param,
ListPrecedent, Platform, TypeParameter. Therefore, it allows handling data, required
by CBR method in our domain.
To develop this database we have selected non-relational (NoSQL) database
management system MongoDB [
In order to estimate an efficiency of the implemented MARS prototype the experiments have been performed using the following scheme: 1) selection of mobile devices for experiments; 2) precedents DB generation; 3) run MARS prototype with different operating modes for ACU (with enabled and disabled ACU).
To test MARS prototype two types of MD were selected, the detailed characteristics of these devices are presented in the Table 1.
In the Fig. 9 presented example of tuple, which describe particular precedents included in the precedents DB. Two series of experiments have been provided with these mobile devices. The first experiment has been provided with disabled ACU (i.e. without adaptation), and the second one with enabled ACU. During these experiments the image resolution of MD screen in case of different values of productivity index (see equation 8) had been measured. In this experiment we take into account two intervals from normal and high ranges of the productivity index: 0.4 R 0.6 and 0.6 R 0.8.The results of these experiments are presented in Table 2 and Table 3 respectively.
The data from the Table 2 show that in case of the fixed image resolution on MD,
and if the value of productivity index R is increased: from the values range [0.4; 0.6]
to the range [0.6; 0.8], then the T(FPS) metric value is decreased, namely, these
values are placed in interval [
Mobile device Nexus 7 Nexus 7 Fly IQ4416 Fly IQ4416 In this paper we have presented the model-based framework to adaptive resource management in mobile augmented reality systems (MARS), which is based on the 3level data structuring and analyzing of their specific hard- and software features. At the conceptual modeling level the appropriate ontological specification of MARS resources was constructed, at the logical modeling level a case-based reasoning approach is utilized, and as a physical model to implement an adaptive resource management in MARS the 3-level reference software architecture is elaborated. This approach was successfully applied in order to solve the task to adaptive management of screen image resolution on mobile device according to changes of its computational loading that finally enabled better video stream quality in MARS.
In future we are going to extend a collection of decision search methodologies in order to improve an adaptation process and compare its efficiency with case-based reasoning approach implementation. Besides that is it supposed to develop a more sophisticated adaptive MARS domain model with wider amount of input and output parameters, which should enable a more configure options in the proposed modelbased adaptive resource management framework.