Extending Augmented Reality Mobile Application with Structured Knowledge from the LOD Cloud

Abstract

ARCAMA-3D (Augmented Reality for Context Aware Mobile Applications with 3D) is a mobile platform that allows us to overlay a 3D representation of the surroundings with augmented reality objects. In this paper, we show how these 3D objects, which are overlaid on the real view captured by the camera of the mobile device, are coupled with the Linked Open Data (LOD) cloud. With this approach, the data aggregation for a mobile augmented reality system is provided using the interconnected knowledge bases on the Web. This offers the possibility to enrich our 3D objects with structured information on the cloud. The objects that act as an augmented reality interface are used to provide an interactive access to these information. This approach provides an opportunity for people who publish information on the LOD cloud to interlink their data with 3D urban models. In order to achieve this, we propose an extensible data model that takes into account the temporal evolution of real world entities (such as buildings, monuments, etc.) and we publish our 3D models using this data model.

1. INTRODUCTION

Ubiquitous mobile applications rely on context and location aware approaches in which the system takes into account the changing context of the user to provide information dynamically. However, constraints related to the small displays of mobile devices and the methods used in order to look for information during the mobility require providing the most relevant information when the user expects a quick answer. For example, in order to discover the surroundings using a mobile application, the systematic and exhaustive presentation of all Points of Interest (PoIs) will not only hamper the readability of these collected information, but also risk to divert the user who observes her surroundings.

In this paper, we present a context-aware Augmented Reality (AR) approach which minimizes the effort from the user to access and interact with the information that may be of her interest. For this, we interconnect our application with the Linked Open Data [ 8 ] which, by structuring the knowledge available on the Web, enables a semantic approach in the information search and discovery.

Augmented reality consists in an interactive medium that overlays the real world with some objects modelled in 3D and keeps their alignment in real time [ 3 ]. By adding a virtual layer on the real world view, the goal is to improve and enrich the user’s perception.

LOD uses the infrastructure of the World Wide Web to publish and link datasets that can then be explored and exploited by applications accessing to them. While achieving some of these objectives still remains a research interest, many datasets from very different domains (media, biology, chemistry, economics, energy, etc.) are already published and are constantly being enriched. A portion of these data contains geo-localized information that can be exploited by the applications, especially while the user is moving. However, few mobile augmented reality applications have explored this possibility.

ARCAMA-3D (Augmented Reality for Context Aware Mobile Applications with 3D) [ 1 ], [ 2 ], our AR system, superimposes transparent 3D representations of the real world objects, with which the user can interact, on the view captured by the mobile device, such as a smartphone. By enriching the 3D models with thematic and temporal metadata published on the LOD cloud, ARCAMA-3D proposes answers to several applicable ubiquitous scenarios in which the augmented reality interface effectively helps the user during her exploration of the real world based on her choices and preferences.

The paper is organized as follows. In Section 2, we present some work that integrates a location-based approach in an augmented reality system, as well as the aspects that differentiate our system from existing ones. Section 3 describes our previous work, the platform ARCAMA-3D and introduces the data model (ontology) that we defined in order to interconnect 3D objects with the LOD. We also show in this section how this interconnection, on the one hand, extends the use of ARCAMA-3D, and on the other, contributes to the expansion of LOD cloud with structured information currently unrepresented in this cloud. In Section 4, we draw a general view of the ARCAMA-3D system, and we conclude in Section 5. 2.

BACKGROUND AND RELATED WORK

The most well-known mobile augmented reality applications are Layar1 and Wikitude2. They provide the user with some local points of interests (POIs) by querying particular servers where such information is stored. The user’s context in these applications is limited to the location and the direction in which the user is moving. Using these information, the databases and servers are queried. In Wikitude, the user discovers information about interesting places, famous landmarks and other POIs. In order to get such information, the application communicates with some servers such as Wikipedia, Twitter, Instagram, world heritage list, etc. Layar integrates information from social networking platforms such as Twitter and Qype, as well as with photo sharing web services such as Flickr. In both applications, the location-based data is overlaid on the real view as an augmented reality tool.

For a person who visits a place, finding and providing all the information by using her geolocation as the unique criterion, may not be considered as satisfactory. The contextaware mobile augmented reality platforms should also consider filtering the information to keep only the relevant information that fits to the expectations or the interests of the user. To this end, Semantic Web technologies can be used to provide a meaningful exploration of the information available on the Web with general and/or specialized knowledge expressed in a formal and structured manner.

Linked data[ 5 ] is a term used to describe a recommended practice to represent, share and interconnect information

1http://layar.com/ 2http://wikitude.com/

using Semantic Web technologies: URIs, RDF, SPARQL. Linked Open Data constitutes a community effort to publish datasets available under open licence respecting the principles of linked data [ 8 ]. Among them, DBpedia, which contains the structured information extracted from Wikipedia, plays a central role in interlinking many other datasets, such as Freebase, LinkedGeoData, GeoNames, Flickr, Revyu, or even FOAF profiles (see Figure 1).

Related to the problem of discovering the surroundings during the mobility of the user, mSpace mobile [ 15 ] is one of the early mobile applications that uses the Semantic Web to support the exploration of information resources. It is a London city guide with connections to different domains or sources of interests (movies, music, etc.). Using the location information of the user, the application queries different knowledge bases (IMDB, video archives of the BBC, etc.) and displays the results in an information box with textual descriptions. Although, this application does not provide any augmented reality experience and its user interface does not allow a fast and efficient exploration, it remains interesting for the use and exploration of multiple interconnected knowledge databases.

DBpedia Mobile, is a context-sensitive browser running on mobile devices [ 4 ]. Designed for the exploration of a touristic place, it displays on a map information about the nearby locations that can be found in the DBpedia dataset. The user can then explore information associated with these locations through links between DBpedia and other data sources (for instance, the associated Wikipedia pages). It also allows users to publish their location, photos and comments on the Linked Data. We note here that the interface does not use augmented reality, but a 2D map which necessitates constant attention of the user to read the information. Moreover, all the information within the vicinity of the user is provided to her without any preliminary selection.

In contrast, in [ 13 ] several possible use cases of linked data are discussed in the context of mobile applications based on augmented reality. [ 14 ] explores the use of datasets related to search and view information about cultural heritage. Using the geolocation and the direction in which the user is moving, the nearby PoIs are identified using the LOD cloud. The authors have focused here on merging and aligning multiple resources associated with the same PoI, as well as the semantic enrichment of the erroneous data (errors in georeferenced data, human annotations, etc.).

In conclusion, these different research projects show the potential of using the Linked Open Data in mobile applications where the information access is based on the geolocation of the user. However, the risk of displaying all the georeferenced information, in particular in augmented reality applications, may expose the mobile user to some cognitive overload [ 10 ]. 3.

CONTRIBUTION

In our previous work, we have proposed a mobile augmented reality platform, ARCAMA-3D, which is based on the use of a lightweight 3D model [ 1 ]. The real-world objects are represented by a semi-transparent 3D geometric model superimposed on the real view captured by the camera of the mobile device (see Figure 2). Thus, the real world is represented by the 3D objects on the user interface. Interacting with these objects, the user can ask questions like “what is it? ”, “Is there something interesting (for me) around me? ” etc.

The ARCAMA-3D application continuously acquires geolocation data as well as orientation data using the embedded sensors (GPS, accelerometer, gyroscope, etc.) of the mobile device (smartphone, tablet PC, etc.) on which it runs. We ensure the alignment of the 3D model on the real scene by refining the captured data using a Kalman filter and merging these data [ 9 ], [ 6 ]. The fusion of sensor data allows an accurate estimation of the position, hence, superimposes the 3D model on the real view, and maintains this superimposition all along the movement of the user [ 2 ].

When we observe users’ demands from an AR mobile application, we notice that they expect a ‘transparent’ augmented reality that should be there only when needed and that should give only meaningful information, following the recommendations made in [ 10 ]. ARCAMA-3D displays the augmented reality objects as transparent, and indicates the objects in different colors if they contain interesting information for the user with respect to a given topic of her interest. By indicating the existence of information in this way, we invite the user to interact with these objects.

The 3D models exploited by ARCAMA-3D play a key role here. They are indeed the support of the augmented reality objects that provide access to information. We propose to represent these 3D models by means of an ontology that captures the characteristics of 3D models (levels of detail, time period that they cover, etc.). Thus, such 3D models can be linked to ontological descriptions of 3D objects by establishing a valuable link between the real world entities described on the cloud. To achieve this goal, we integrate the geometric descriptions (3D models) of a realworld entity corresponding to a nearby Object of Interest (OoI) [ 1 ] of ARCAMA-3D in an RDF graph. This graph uses a controlled vocabulary defined in an OWL ontology (arcama-owl). This ontology defines a generic model for ARCAMA-3D; and the RDF graph corresponding to a given entity can be seen as an instance of this generic model.

The representation of 3D models used by ARCAMA-3D brings two advantages: • it allows to connect - and thus enrich - these 3D models with other structured datasets (especially Freebase, DBpedia, etc.) • symmetrically, it offers the possibility to other datasets of the LOD cloud to bind their information with 3D models and re-use them. 3.1

We have created a 3D data model where we store the geometric representations of real-world entities and the information associated with them. Our data model is extensible, and the representation of an OoI can be associated with temporal and thematic information. Also, in the design of a 3D data model, we think that it is necessary to incorporate spatial (what portion(s) of space the 3D model covers ), temporal (which time period(s) the 3D model belongs to) and thematic (what is (was) the role (function) of that real world object) characteristics in the model. We adopt a similar approach to that described in [ 11 ] which proposes a spatio-temporal conceptual model focusing on the orthogonality of spatial, temporal and structural instances. In this work, the authors apply this approach to non-urban geographic features (rivers, lakes, townships, etc.) or 2D spatial data relating to urban objects (the surface of a castle, etc.). We integrate this approach in our 3D data model to allow the exploitation of spatial, temporal and thematic information related to the OoIs during the AR experience of the user.

To illustrate this model of 3D data using a concrete example, let us consider a real world entity: the Hagia Sophia edifice in Istanbul, Turkey. The choice of this entity is suitable to illustrate our approach, since the architecture and the role (the function) of the building have both evolved over time. Thus, on the Wikipedia page dedicated to this monument3, we can find in the infobox (a table that summarizes the article on the right corner of each Wikipedia page), the information related to this evolution.

The DBpedia resource representing Hagia Sophia4 assigns successive roles to it as eastern orthodox cathedral, mosque and museum (see Figure 3). However, no indication related to the relevant time periods of these functional changes nor its architectural changes is presented in this description. The purpose of our data model is to fill these gaps. Thus, the proposed model takes into account both the temporal evolution in space (geometry) and thematic information (role/function) that describe the OoIs. Therefore, for a given OoI, we need multiple representations of the object, each of them corresponding to a change in the thematic and/or the geometric dimension. Figure 3 shows the evolution for the Hagia Sophia, and the different representations (i.e. R1, R2, R3, R4 in the figure) that the model should be able to hold for this OoI. As expected, each of these four representations correspond to an architectural change (geometric) and/or a functional change (role) of the OoI. 3.2

Arcama-owl Ontology

In order to construct these multiple representations, we have defined an OWL ontology that describes a generic model for an OoI. The UML class diagram in Figure 4 shows the different concepts (classes) and relations between these concepts (properties) defined in this ontology.

For our system, this class model (and its instantiations) plays an important role: it allows the interconnection between the historical 3D models associated to OoI and the information related to these OoIs found on the LOD. • The Entity class corresponds to an OoI (the real-world entity Hagia Sophia, for example). This model al

3http://en.wikipedia.org/wiki/Hagia_Sophia/ 4http://dbpedia.org/resource/Hagia_Sophia/

lows to combine several temporal representations of the same entity (see class TemporalRepresentation). The represents association allows to interconnect this model with other resources described in other datasets and published on the LOD cloud (for instance, the DBpedia resource5 or the Freebase resource6 for the Hagia Sophia). • The TemporalRepresentation class is at the heart of our model. It corresponds to a given representation of the entity. This representation has a temporal validity interval defined by OWL-Time7 with hasTimePeriod association. A temporal occurrence aggregates spatial and thematic attributes that describe the entity during this time interval. These attributes are described respectively by Geometry and Role classes. Any changes in the Geometry and/or Role information require creating another TemporalRepresentation associated with a new time interval. • The hasRole association sets the role(s) of a TemporalRepresentation object through the Role class which holds the properties about the function of the object (museum, monument, hall, church, etc.). This information is used to classify the temporal representations according to the thematic criteria chosen by the user. Accordingly, it allows to filter temporal representations that will be displayed as the augmented reality interface and allows the access to additional information related to the that entity. All the roles presented here are defined by another ontology that we have created and currently has 125 classes corresponding to different types of architectural structures, for example, religious buildings (cathedral, mosque, temple, etc.) or historical sites (castle, monument, pyramid, etc.). To facilitate the interconnection with resources from other datasets defined with their own ontology, a mapping between roles and classes of some of these ontologies is defined (currently with DBpedia and Freebase datasets). Initially, we had planned to base it solely on the roles used in the DBpedia ontology, but it turned 5http://dbpedia.org/resource/Hagia_Sophia 6http://www.freebase.com/m/0br5q 7http://www.w3.org/TR/owl-time/ out, on the one hand that the proposed DBpedia structure was not fully satisfactory, and, secondly, that the roles of the resources could be of variable qualities (i.e. depending on the quality of the information provided by Wikipedia contributors). Using our own ontology and establishing a mediator mechanism allowed us to free ourselves from these drawbacks. • In order to describe the geometry associated with a time instance, we use the composite design pattern [ 7 ] that combines geometry in a tree structure that represents a part-whole hierarchy. Geometry is an abstract class with two subclasses ElementaryGeometry and CompoundGeometry. • The CompoundGeometry class allows to define a complex geometry composed of several different geometries that can be either complex geometries or elementary geometries. • The ElementaryGeometry class describes a leaf node in a tree structure of geometries. As for a specific purpose, we want to be able to offer different 3D models at different levels of detail, ElementaryGeometry class aggregates one or more GeometryFile objects. Each GeometryFile object is described by a URI that provides access to a physical file containing a description of the 3D object in a usual format (KML, CityGML, VRML, COLLADA etc.). • The LevelOfDetail class describes the level of detail of a GeometryFile. We have defined three levels of detail where LOW corresponds to a rectangular prism. It corresponds to the LOD18 of CityGML. MEDIUM corresponds to a much detailed 3D geometry with no texture. HIGH corresponds to texturized 3D objects (it can even represent a laser scanned object).

Figure 5 presents a part of the RDF graph corresponding to an instantiation of this model for the example of Hagia Sophia. In order to facilitate the interpretation of this RDF graph in relation to the generic model presented above, we 8Please note that this acronym is not related to the Linked Open Data appreviation (LOD) that we often use in our article. choose to color and format the resource nodes that belong to the same arcama-owl class in Figure 4 with the same color and line format.

In this RDF graph (Figure 5), Hagia Sophia is represented by two temporal instances: model:HagiaSophia#TI1 and model:HagiaSophia#TI2. The former one is the instantiation of the model in interval R2 of Figure 3 and the latter one is of R3. As it can be seen in the temporal evolution of the real world entity, first temporal instance represents a change in the geometry of the entity, whereas the second one represents the changes both in the role and the geometry.

ARCAMA-3D

The mobile device (ARCAMA-3D client) connects to the ARCAMA-3D server (see Figure 6). Using geolocation information and user thematic preferences (architecture, history, museum, etc.), the server queries the triplestore containing descriptions associated with 3D models using SPARQL. Then, the corresponding 3D geometries are superimposed on the actual view in a semi-transparent form, and in different colors if they have some results from the SPARQL query associated with them. The user can then interact with the 3D objects by selecting the object on the screen of the mobile device (3D picking technique). This interaction gives access to information related to this object in the LOD cloud (photos interlinked with that object on the LOD cloud, historical 3D models and roles of the buildings during different time periods, etc.). Then the user accesses these information. 5.

CONCLUSION AND FUTURE WORK

New datasets are continuously being added contributing to the evolution of the LOD cloud which is a semantically interconnected knowledge base. We describe in this paper a system dedicated to ubiquitous mobile applications based on augmented reality which includes temporal, thematic and spatial representations to interlink 3D models with the information available on the LOD cloud. The interconnection with the LOD cloud not only helps to represent 3D datasets on the cloud, but also improves the integration process of the augmented reality system. Temporal aspects supported by the proposed model reflect the evolution of real-world objects in their form, function, location, etc.

We now intend to improve the representation of the identity of objects in the real world and the characterization of their evolution over time, adapting some of our previous work [ 12 ].

ACKNOWLEDGMENTS

This research is granted by the Region Rhoˆne-Alpes.

References

[1] B. Aydin , J. Gensel , S. Calabretto , and B. Tellez. Arcama- 3d - a context-aware augmented reality mobile platform for environmental discovery . In W2GIS , pages 17 - 26 , 2012 .

[2] B. Aydin , J. Gensel , S. Calabretto , and B. Tellez . Arcama-3d: Une application mˆelant r´ealit´e augment´ee et 3d pour la pr´esentation d'informations contextuelles. In 8`emes journ´ees francophones Mobilit´e et Ubiquit´e (UBIMOB), Anglet , France, pages 34 - 41 , 2012 .

[3] R. Azuma . A survey of augmented reality . Presence , 6 ( 4 ): 355 - 385 , 1997 .

[4] C. Becker and C. Bizer . Exploring the geospatial semantic web with dbpedia mobile . J. Web Sem ., 7 ( 4 ): 278 - 286 , 2009 .

[5] T. Berners-Lee and M. Fischetti . Weaving the web - the original design and ultimate destiny of the World Wide Web by its inventor . HarperBusiness , 2000 .

[6] R. G. An intelligent multi-sensor system for pedestrian navigation . Journal of Global Positioning Systems , 5 ( 1 -2): 110 - 118 , Sept. 2006 .

[7] E. Gamma , R. Helm , R. Johnson , and J. Vlissides . Design patterns: elements of reusable object-oriented software . Addison-Wesley Longman Publishing Co., Inc ., Boston, MA, USA, 1995 .

[8] T. Heath and C. Bizer . Linked Data: Evolving the Web into a Global Data Space . Morgan & Claypool, 1st edition, 2011 .

[9] R. E. Kalman . A New Approach to Linear Filtering and Prediction Problems . Transactions of the ASME - Journal of Basic Engineering , ( 82 ( Series D )): 35 - 45 , 1960 .

[10] T. Olsson , T. Ka¨rkka¨inen, E. Lagerstam, and L. Venta ¨ -Olkkonen. User evaluation of mobile augmented reality scenarios . JAISE , 4 ( 1 ): 29 - 47 , 2012 .

[11] C. Parent , S. Spaccapietra , E. Zima´nyi, P. Donini, C. Plazanet , C. Vangenot , N. Rognon , and P. A. Crausaz . MADS, mod `ele conceptuel spatio-temporel . Revue Internationale de G´eomatique, 7 ( 3 -4): 317 - 352 , Dec. 1997 .

[12] C. Plumejeaud . Mod`eles et m´ethodes pour l'information spatio-temporelle ´evolutive . These, Universit´e de Grenoble, Sept. 2011 .

[13] V. Reynolds , M. Hausenblas , A. Polleres , M. Hauswirth , and V. Hegde . Exploiting Linked Open Data for Mobile Augmented Reality . In W3C Workshop: Augmented Reality on the Web , June 2010 .

[14] C. Van Aart , B. Wielinga , and W. R. Van Hage . Mobile cultural heritage guide: location-aware semantic search . In Proceedings of the 17th international conference on Knowledge engineering and management by the masses , EKAW'10 , pages 257 - 271 , Berlin, Heidelberg, 2010 . Springer-Verlag.

[15] M. Wilson , A. Russell , D. A. Smith , A. Owens , and M. C. Schraefel. mspace mobile: A mobile application for the semantic web . In User Semantic Web Workshop, ISWC2005 , 2005 .