This poster and accompanying demo shows how Semantic Web and Linked Data technologies are incorporated into an Augmented Reality platform in the SmartReality project and form the basis for enhanced Augmented Reality mobile applications in which information and content in the user's surroundings can be presented in a more meaningful and useful manner. We describe how things of interest are described semantically and linked into the Linked Open Data cloud. Relevant metadata about things is collected and processed in order to identify and retrieve related content and services on the Web. Finally, the user sees this information intuitively in their Smart Reality view of the reality around them.
SmartReality is a project which began in October 2010 as a nationally funded project in Austria. The participating partners are STI International, Technical University of Graz, Seekda GmbH and play.fm GmbH. Together they represent expertise and innovation in Augmented Reality (AR), semantic technology, Web services and online media. In the context of the project, they explore how people may be able to access relevant information, content and media about things of interest in their vicinity via AR. The AR is enhanced dynamically by the Web of data and the Web of services. This enhancement is made possible by a SmartReality platform which mediates between the client device and the Web-based data and services, focused on using available metadata to select the most appropriate content and services for display to the user. This poster and accompanying demonstrator will present the first results and prototypes of the project. We will explore and demonstrate how semantic technology and Linked Data can be integrated with AR in order to make a user’s reality “smarter”, based on a scenario with street club posters. 2
The value of the ideas of Semantic Web and Linked Data to the AR domain has been
reflected in recent position papers, not only from SmartReality [
For our initial application domain, SmartReality is focusing on music. Music is a common and increasingly shared experience via the Internet. The user group most likely to be early adopters of Smart Reality solutions are young professionals who are typically interested in listening to music, discovering artists and attending concerts. Together with our partner Play.fm GmbH, whose web site and mobile apps offer access to more than 18 000 DJ mixes and live recordings to 150 000+ users a month, our goal is to use semantics and Linked Data to dynamically link references to music around us in our current reality to virtual information, content and services from the Internet which enhance our experience of music in our current reality. In the SmartReality use case, for example, we consider a music-conscious person wand ering city streets and seeing the ubiquitous street posters advertising concerts and clu b nights. Now, at best, if this person is interested in the content they may have mobile Internet and can begin to search on aspects like the artist, event or venue. However, this presupposes they can identify these aspects from the poster and they still must g ather and link together the information they are interested in across various Web searches, e.g. find out about the artist, find some audio to listen to from the artist, check wh ere is the venue where the artist is playing, find out how to get there, find out how to get tickets for the concert. However, with Smart Reality they can view an enriched version of the poster with related content automatically overlaid over the poster references to artists, events, venues and other things. This is illustrated in Figure 1. 3
A SmartReality server handles the interaction between the client and the Web of data and services. A simplified illustration of the steps taken in SmartReality is given below (Fig. 2). First, the object in the mobile devices camera view is identified via an image recognition service (we use Kooaba1), which returns an identifier for the object. This identifier is linked with a description of a “Thing of Interest” (TOI) in a datastore we term a “TOI Repository”. The TOI description, enriched by links to concepts in the Web of Data, is processed in order to select the most appropriate co ntent and services from the Web. The resulting content is packaged and sent to the client for display in the AR view. 3.1
To ease the process of generating the initial metadata about Things of Interest (TOI), we implemented a Web based annotation tool (Fig. 3). The tool currently only supports selecting street posters from play.fm’s image database. Here, the user selects regions for triggering the appearance of content as well as regions where to actually display the content over the poster. Instead of adding a concrete link to con tent from the selected regions, users rather select a LinkedData URI representing a concept from an existing conceptual scheme. In this case we use the Linked Data identifiers for play.fm (artists, events, clubs) and support their addition by allowing free text entry and Ajax-based concept selection (automatically filling in the full U RI of the concept). The user is also free to use a full URI from any other Linked Data source. When editing is finished, the annotation tool generates a Thing of Interest (TOI) annotation for the poster and stores it in a TOI repository. Additionally, the image of the event poster is uploaded to Kooaba to make it possible to identify the poster at runtime. The TOI data model has been created in RDF/S specifically for SmartReality and is published at http://smartreality.at/rdf/toi. 3.2
The server is developed as a set of components which interchange messages to realize the SmartReality functionality expressed in the above workflow (Fig. 2). The platform has been developed in Ruby on Rails and exposes a RESTful API to clients. The repositories and APIs used by the components to retrieve data into the workflow are separated from the code of the core components so that different storage and remote API solutions (including cloud) could be used as required. After parsing the TOI’s metadata (the TOI being identified via the Kooaba identifier which is included in its description in the repository), it provides an initial response to the client which identifies the TOI’s regions of interest for display in the AR view (see below, Fig. 4 left). Two further functional steps are realized on the server to provide the content bundle for the enrichment of the TOI’s regions of interest in the AR view with links to content:
• Linked Data consumption. The (Linked Data) concepts used to annotate the TOI’s regions and extracted from the TOI’s metadata are crawled and further, related concepts extracted as defined in a set of Linked Data crawling rules. The rule syntax makes use of LDPath2. As a result, a local repository of relevant structured metadata about the concepts of interest in the TOI has been created. We use mainly play.fm Linked Data3 in the current demo, while the approach is vocabulary-independent, i.e. any Linked Data could be used in the annotation and supported in this step. For the Linked Data step, we make use of the Linked Media Framework (LMF4), which provides on top of a RDF repository the means to directly cache Linked Data resources and synchronize their metadata with the description on the Web. This means for a new annotation the LMF will automatically use the locally cached resource metadata in available rather than repeatedly retrieve it from the Web which can lead to latency in the platform response.
• Service selection and execution for content retrieval. Based on this local
metadata, a service description repository is queried. Services or APIs are described in terms
of their conceptual inputs and outputs so that, for the given class of a concept in the
annotation, appropriate services can be found to provide content for the enrichments
of the TOI in the AR view. In the current demo, we use an API provided by play.fm
to access audio streams of recordings by artists as well as an API provided by Seekda
to link an event to a booking interface for nearby hotels with rooms available on the
night of the event. Service execution will require querying the concept descriptions to
extract the necessary input values – e.g. for the hotel booking interface, the service
API needs the event’s data and its location’s longitude and latitude to be passed in the
input request. Likewise, the service response needs to be parsed to return to the
SmartReality platform the content inks which can be used to enrich the TOI with respect to
the original concept. For this, we use the concept of “lowering” and “lifting” in the
Linked Services approach [
output response are ‘lifted’ to new semantic concepts (e.g. from a Place to images of Maps of the place).
The “lifted” responses are collected and sent as a content bundle to the client in JSON. 3.3
We built a client application prototype for Android smartphones. It lev erages an Augmented Reality interface visualizing relevant information that is spatially registered on the physical object via Natural Feature Tracking. A user p oints her smartphone on the physical thing of interest to initialize a TOI query. After successful initialization, segments containing relevant information are highlighted th rough an Augmented Reality interface on the physical object (Fig.4 below left). The user can now point towards individual segments and obtain detailed information (Fig.4 below right). The rendering of content that is spatially registered to segments on the physical object in 3D space is based on OpenSceneGraph5.
The SmartReality demo will use two real club event posters with the installed client on Android smartphones to give visitors the experience of SmartReality for themselves. The SmartReality project has focused on a proof of concept with club event posters and enrichment via LOD from mainly the play.fm database. The infrastructure developed has been deliberately designed to separate distinct data and content sources from the workflow which realizes a SmartReality experience, i.e. the use of other objects as “Things of Interest”, the annotation with other LOD sources, or the linkage to content from other providers for display in the AR view, should be feasible as a configuration issue and not require any changes to the SmartReality platform or client. 4 5
6