-Agents based application should be able to use pervasive sensors as their own perceptors in order to let the users benefit of context awareness in services exploitation and in order to make mobile services really useful and profitable. The agents described in our approach adapt themselves in response to a changing environment and are able to respond differently, creating entirely new plans, and thus changing their own behavior depending on their past experience. We present a framework that implements the model to supports experts in the domain of the Cultural Heritage to augment the archaeological site with a set of multimedia contents which are delivered by intelligent services to the visitors in order to guide their tour and to enhance their perception of the reality.
Pervasive environments are really dynamic and enable
software applications to access to large amounts of
information from anywhere. However, context awareness involves
capturing and making sense of imprecise and sometimes
conflicting data and uncertain physical worlds [
In this area the tourist guide plays an important role because of the increasing of more exigent audience leading to a request of more professional services for any kind of tourist which best fit his needs and desire of gaining satisfying and personalized information. Personalization and adaptation are a key concept in this field because it is really difficult for a mobile tourist in a dynamic environment to browse and search desired information and services. In this kind of contests, personalization and adaptation allow the system to became a sort of intelligent personal assistant characterized of being proactive and autonomously providing suggestions and support.
The agent-oriented paradigm is highly suited for applications dealing with complex and dynamic environments, that are usually unpredictable (it is not possible to predict the future state of the environment), unreliable (the action of the agent can fail because of factors beyond its control).
In [
In particular BDI agents are a type of intelligent agents characterized by flexibility with particular mental attitudes based on human concepts, such as beliefs, goals and intentions.
In BDI architecture [
Beside intelligence is also provided from the capacity to quickly adapt its behavior so an intelligent agent needs knowledge about the environment to take decision finalized to reach the goal. In fact an agent can be defined intelligent if has a Knowledge Base. The elaboration of intelligent plans needs a basic knowledge for interpreting perceptions and for relating them to previous ones, to infer new or hidden knowledge combining previous and new perceptions. The capacity of reasoning of an intelligent agent is directly proportional to the information included in the knowledge base (KB), so is desirable to have a wide KB.
In our approach an intelligent agent executes on the
user’s device to support services exploitation. It percepts
the surrounding environment using the on-board peripherals
and executes plans which are chosen by an ad-hoc reasoning
to optimize the user’s satisfaction. Besides as the device is
a PDA (Personal Digital Assistant), it has limited battery,
storage and computation capacities so also the agent running
on it has limited capacities. In order to extend the agent’s
capability another remote intelligent agent interacts with
the local one to achieve individual or shared goals. This
two-agent system is part of a loosely coupled network of
problem-solver entities that work together to find solution
to problems that are beyond the individual capabilities or
knowledge of each entity [
The agents described in our approach are developed using Jadex 2.0 and adapt themselves in response to a changing environment and are able to respond differently, creating entirely new plans, and thus changing their own behavior depending on its past experience. Agents also have a library of predefined plans that are invoked by events like the add or removal of a belief, the reception of a message, and the add of a desire. Besides each plan might consist of several subplans to achieve the intention. In fact, to satisfy a specific desire, it is necessary a plan that allows to reach that goal starting from given beliefs.
The interaction between the two agents has been modeled according to a learning by teaching approach: a student on the device and a teacher in remote. The student agent learns from the teacher and the learning process can be started from both of them. As PDA has limited capacities the teacher give to the student a new plan directly and it will be stored in the agents plan library. The common goal is the maximization of the expected satisfaction of the user.
The first thing to be considered when modeling this kind of problem is the environment. In fact we have to model the environment where the user is moving and to reconstruct the perceptions of the user himself in order to get his particular vision about what is surrounding him. A real representation of the environment is necessary to identify landmarks and possibilities of intervention using pervasive actuators and sensors whose input will be stored in the memory of the agent and will be updated as the environment changes. The knowledge of the environment acquired by the agent represents part of its own beliefs.
In Figure 1 an high level representation of this model is shown.
The environment will be modeled as a geo-referred map with itineraries, landmarks and points of interest. Of course localization of users and objects is possible according to the device technology, the available infrastructures and the kind of environment. Indoor or outdoor localization can be implemented using heterogeneous technologies, and often absolute localization could not be performed, but only nearby landmarks or object can be detected.
In our model another set of believes describes user’s knowledge about the surrounding environment that can be acquired by using peripherals of his personal mobile device, by recording and evaluating user actions or explicitly asking for user feedbacks. Some examples are user’s position, interest, nearby objects, landscape, etc. Interaction with the environment and presentation of contents can be done again using user’s device and his peripherals.
Of course different components of the user’s knowledge could have different weights according to their relevance or the time of perception. We could consider the knowledge at the current time, in a time window, or the complete history.
Finally another set of believes is composed of contents in the knowledge base of the teacher agent. But to maximize the user’s satisfaction it is relevant the way they will be delivered, to be used for augmenting the reality perceived by the user. Services, in order to augment the user’s knowledge and the their/user’s capability to interact with the environment, have to choose, according their context awareness: what content and application should be delivered; when it needs to execute the application or to present a specific content; how this should be done.
Plans are queries in knowledge base that select the optimal set of contents and organize them according to user’s profile, device and time availability. The optimal set, and its organization, is chosen to optimize the user’s satisfaction. The maximization of user satisfaction represent the desire to reach.
IV. ONTOLOGY BASED KNOWLEDGE REPRESENTATION
AND REASONING
In a multi-agent system agents communicate among them in order to reach the global goal or their local goals but they can disagree about what that knowledge means. For this reason, ontology is used to enable knowledge sharing among different agents and to improve the quality of the service provided by agents.
An ontology implements the representation of the global knowledge. It is necessary to share a common dictionary and to describe the relationships among the entities/objects which are part of the model. In our model a common ontology includes all the general concepts which are useful to describe a pervasive environment where mobile users are moving, using their devices and interacting with available facilities and other users. The general ontology is complemented with a domain ontology that is designed by an expert of the specific application field. Concepts of the ontology are used on client side to describe a representation of the reality as it is perceived by the user. On the back-end the ontology is used to annotate digital resources like POI (point of interests), contents, applications. It is also used to support reasoning. Each agent knows how to translate its knowledge into the common ontology before sending a message and after receiving a message from another agent. Users’ behaviors, information from pervasive devices or from other users, device properties, external events are heterogeneous data are used to build a dynamic changing representation of the user knowledge about the reality within which he is moving. The applications are knowledge driven. The user’s knowledge can be used by the application that is running on the device to adapt its logic locally and is updated remotely to improve the awareness of services at server side.
Applications are events based. Events can be updates of the user’s knowledge or can be explicit service requests raised by the user. At each invocations some semantic queries, that depend on the user’s knowledge, are built and processed to get the action to be performed and the contents to be delivered. Results of the query are individuals of the ontology that are described by semantic annotation.
The user’s knowledge is composed of many semantic concepts with static, dynamic or locked properties.
Components of the knowledge are:
properties here we means hardware resources, on board peripherals, display size, total memory and storage. Dynamic ones can be power level, available memory and storage and bandwidth ; User’s position. It is a dynamic property that can change over the time and can be evaluated using different techniques and technologies, depending on the devices, on the available infrastructures and the kind of environment (indoor or outdoor); Pervasive objects. They are dynamically discovered by the device. They can be sensors which provide information about the environment or actuators. They can be accessed and used by the services themselves using the device or eventually using the network if pervasive objects are directly connected; Time information. We intend the current time at user side and the time that the user is spending, or has spent within the environment.
User’s interests. This part of the knowledge could be dynamically changed by the application according to user’s behaviors and to his feedbacks. The user could choose to start from an empty or a standard profile, to change it or to lock some properties interactively.
Semantic techniques are used for intelligent content and application discovery and delivery. Knowledge representation, ontology and annotations of digital resources are used to filter, organize and deliver contents and software to the device. Different techniques for reasoning can be experimented such as graph matching, description logics, neural networks or more simple ones like SPARQL (SPARQL Protocol and RDF Query Language) queries. Furthermore semantic can be integrated with other kinds of techniques to take into account constraints such as user’s position and available time for exploitation.
Users, agents, services and producers are actors in this scenario.
On the left side the user is using his device that hosts an agent that is able to perceive information from the field by pervasive sensors.
The agent student executes autonomously and proactively in order to support the users’ activity within the environment he is moving. It discovers surrounding objects, uses them to update the representation of user’s knowledge, reacts using the local knowledge to organize and propose the available contents and facilities by an interactive interface. It could also communicate with close devices. If the connection works the device can access remote services which can exploit a wider knowledge and reasoning capabilities to look for additional contents and applications.
The agent teacher runs different reasonings, according to user’s profile, that make queries to the Fedora1 repository and returns the selection of contents that best fit user’s needs of information. Teacher’s beliefs are the ontology, a repository of semantically annotated content, and the beliefs sent by the agent student. During the deliberation process a SPARQL query is generated. The SPARQL client submits it to the SPARQL engine of the Fedora repository in order to collect a set of structured and organized media contents representing the plan that student agent has to execute. The execution of a plan by the student agent supports the user in the exploitation of the pervasive environment by augmenting his itinerary with media contents. The agent student has a limited intelligence so his task is to organize and propose the available contents according to its beliefs.
In Figure 3 is described the interaction between agent student and teacher student. Experts of the application domain define the ontology 1http://fedora-commons.org/ for the specific case study. They use or design a map to represent the environment. They add POIs to the map to georefer multimedia contents and can link them to a concept of the ontology. Furthermore they select relevant contents and annotate them using concept and individuals from the ontology.
Remote applications implement context aware services. They use personal devices to collect perceptions and for content delivery.
In order to support these activities in a real scenario we need to provide: back-end tools for enabling content production, their semantic annotation publishing and retrieval; services for content discovery, adaptation and delivery; a client application to support user interaction with services, with the environment and with available contents.
Exploitation of archaeological sites can be very difficult because of a lack of supporting infrastructures and because of the complex recognition and comprehension of the relevant ruins, artworks and artifacts. The availability of personal devices can be used to plan and support the tourist by suggesting him the visit tours, the point of interest and by providing multimedia contents in the form of digital objects which can semantically augment the perceived reality.
In this context a relevant issue is the profiling of the
user, the selection and the presentation of the contents
which can improve the user’s satisfaction, by providing new
models of interactions with reality, trough his device. In
this context the Second University of Naples is engaged
on a multidisciplinary project with both cultural and a
technological aims [
Following the approach defined above we are implementing a technological framework that supports the experts in the domain of the Cultural Heritage to augment the archaeological site with a set of multimedia contents which are delivered by innovative services to the visitors through their mobile devices in order to guide their tour and to enhance their perception of the reality and learning.
Three case studies have been chosen to test the approach and the framework. The S. Angelo in Formis Basilica, in Campania, near S. Maria Capua Vetere, the ancient town of Norba and on the amphitheater of Capua.
To satisfy the requirements of the presented case studies we need to provide a technological solution that does not need infrastructures for letting the software know the user location and his feeling about the environment. It means that Bluetooth, RFID (Radio Frequency Identification), GPS (Global Positioning System ), electronic compass, camera, network connection and others are the technologies which can be used together or independently to get information about the user perceptions and to augment his exploitation of the archaeological site. The user will be able to download at home, before to leave, or on site, if the network will be available, the map of the area to be visited. The map will include all the points of interest that identifies the relevant objects of that area and different cultural itineraries which could be exploited on site. Also contents can be discovered and downloaded in advance. On board software and remote services will assist the cultural visit by augmenting the reality by the user’s personal device. In the following we detail the technological choices which have been taken to implement each component of the framework.
To provide a description of the environment within which the user is moving we need a geo-referenced map that describes buildings, roads, bans, itineraries and Points of Interest. We used the OpenStreetMap format to design open maps. In Figure 4 the map of the S. Angelo in Formis Basilica is shown. It has been built by exporting a model that was originally built by Autocad and has been exported into a GPX format. The JSON (JavaScript Object Notation) tool allowed us to import the GPX trace and to add manually details and POIs. Each point represents an artifact, a ruin or any other entities of cultural relevance and can be described using a list of key-value pairs. Some of them have been used to link the POIs to URLs of multimedia information, or to provide a semantic description of the POIs itself. The tool allow to export the map in an open format that can be read and used by the client application that is presented in Section VI.F.
An ontology has been designed to describe the sites of interest and to annotate the related media. A general part include the concepts which are common to all the class of applications that can be modeled according the proposed approach. Among the others the Time class and his properties (CurrentTime, AvailableTime, ElapsedTime, ExploitationTime) allow to organize and assist the visit taking into account time information and handling time constraints. Position class and its properties allow to localize the user and objects around him. An application specific part of the ontology include the concepts that belong to the domain of the cultural heritage and additional classes and individual which are proper of the case studies introduced in the previous section.
The ontology is used also for annotating the multimedia contents. To annotate texts, images and any kind of contents we chose the AktiveMedia tool2. In Figure 5 a picture of the Amphitheater of S. Maria Capua Vetere is annotate with the Column and the Arc classes which are part of this kind of building.
2http://sourceforge.net/projects/aktivemedia/).
The output produced by the annotator is an RDF file that use concepts and properties of the AktiveMedia ontology and of the domain ontology.
The Fedora repository 3.4.2 is used to store digital objects and supports their retrieval. Into the Fedora repository a digital object is composed of a set of files which are: object metadata: used by the client application to understand how to deliver the content; binary streams: which are images, video, text ... any kind of raw information to be delivered; RDF annotation: that describe the semantic of the object according to the ontology; disseminations: filters to be eventually used for adapting the object according to the target client.
We loaded the Aktive-Media ontology and the domain ontology into the Fedora repository in order to exploit its embedded SPARQL engine that is used to select the optimal set of individuals that means contents. Multimedia contents are automatically stored into the repository after the annotation phase. The RDF output is automatically processed using an XSL transformation to make it compliant with the format required by the Fedora repository.
Different types of content models have been defined and simple examples have been produced.
Multiple images whose transparency can be graduate by the user to compare changes in different periods. In the same way real picture can be compare with paintings. Old picture can be compared with what is seen by the camera.
Part of the image acquired by the camera are recognized and linked to related multimedia contents; Virtual reconstructions which are synchronized with the camera output or the detected RFIDs;
Text, audio, video and composite media.
A content descriptor is attached to every digital object. It is used by the device when the content must be delivered. The descriptor defines the right player for that media, configuration parameter and necessary input.
In Figure 6 an example of delivered content is shown. The user focus the camera on a particular view of the S. Angelo in Formis Basilica and sees the original temple of Diana Tifatina. In particular the perspective viewed by the camera is the same shown by the video.
A remote service has been conceived to support content discovery and delivery.
It uses multiple criteria to discover and filter relevant multimedia contents. Some criteria are the available time for the visit, the user’s position, the device technology, the user interests. The service is implemented by independent filters. Filters are executed on the occurrence of such event that update the user knowledge, or by direct asynchronous request from the user himself.
For example a change of the user position is used to filter POIs which are close to him or are relevant to his position. For example we could get information about a building located close to the user or about some other building located elsewhere but which have been designed by the same architect.
When the user shoots a new picture, a search by sample facility is used to find the image which are similar to the the current subject. This can be used to suggest a new set of contents which relevant to what the user is looking at.
Each time the interest of the users changes, or also when any other event occur, a reasoner generates dynamic SPARQL queries, by which the repository is searched, and organizes the retrieved contents according to their relevance. The ontology is used to perform additional reflexive reasoning.
Additional filtering rules can be implemented using time information if they are available. For example it could be relevant to know the current time (to suggest a sunshine rather than to avoid a closed attraction or museum). The available time for the visit is important to limit the number of contents and to plan an itinerary. A delay is important to dynamically cut what is less relevant and cannot be exploited anymore.
We implemented a search by sample filter that exploit the position of the user to select all the picture which represent the subjects he shoot, such as landscapes, monuments or buildings around him. Within this set a matching algorithm is able to identify an image that is the most similar to the picture. All the annotation for that image are also matched with on the display of the user’s device.
Let us suppose that one of agent’s goals is to augment the visitor’s itinerary with multimedia streams, supported by his device and relevant to his interest. We have to search for all multimedia contents that are compliant with the encoding supported by the user’s device and describe the specified architectural element of a particular historical period. An example is provided in the following: PREFIX device: <info:prist/profile/device#> PREFIX prist: <http://www.dcs.shef.ac.uk/˜ajay/image/annotation#> SELECT ?a FROM <#ri> WHERE {?a prist:hasConcept "Arcata"}
OR {?a prist:annotationText "Arcata"} AND {?a prist:hasInstance "96-217_DC"}
AND {?a device:hasEncoding "mpeg"}
Obviously the list of retrieved objects will be also filtered and ordered according to other criteria like time availability and semantic relevance.
At client side we extended an open source software navigator called Navit 2.0 3. We extended the android version. The navigator provides basic facilities for map visualization and to guide the user along some pre-defined itineraries by using the on-board GPS receiver. By new extension the guide is able to sense the environment by the available peripherals, to understand the situations according to which the visit will be adapted, and to enhance the user experience. Even if many experiences on augmented reality are proposed nowadays, the exploitation of vision is used only to overlap real and virtual images. The extensions provide the following functionalities: access to device peripherals to sense the environment and to update the representation of the user’s knowledge about the reality around him; access to remote services to update the user knowledge and to ask for available contents a local cache of objects for exploiting the visit without connection; a limited reasoner that is able to organize the content by itself when the connection does not work.
GPS positioning to localize the user in open spaces and to guide him on cultural itineraries; RFID for positioning and detection of nearby POIs. This technology can be used to alert the user but also for his positioning in indoor environments; CODBAR recognition to get information about artifact, monuments, ... when a RFID reader is not available; image recognition by search by sample techniques which are speed-up and improved using a position based filtering; monitoring of device resources and configuration; collection of user interest by feedback and by an analysis of his behaviour; time monitoring 3http://www.navit-project.org/
On the other hand a list of functionalities are provided to deliver contents and to guide the visit. Content management (discovery client,organization and fruition) is supported both at client side and at server side. Knowledge visualization and management to allow the explicit specification of user’s own interest is provided. The output of the camera is used as a component of the user’s knowledge as well as a map on which semantic additions are anchored, not simply superposed.
The user will be able to ask for and exploit available multimedia contents, which are related to points of interest, or to personal interests expressed by semantic concepts. Multimedia content will be adapted at server side according to the device/user/session profile to provide to the user the best quality of service.
Figure 7 shows the output of the result of the content discovery service. The service invocation starts the camera that is used to take a photo of the landscape or of a subject of interest. Image upload, discovery of relevant contents and download of retrieved information run in background meanwhile the user is interacting with other facilities provided by the client. The list of contents is presented to the user when it is available. Other events which open dialog for suggesting action, itineraries, POIs or media can be related to new perceptions.
Many research contributions focus on the development of
software/hardware architecture and frameworks for mobile
context-aware tourist guide, also based on agent technology
and semantic techniques. In [
Beside pervasive technologies and augmented reality are used to enhance the visit experience and enabling visitors to explore easily the museum according to their interests.
In this paper we presented an approach for exploitation of intelligent agents technology in order to create and deploy various ubiquitous services for a mobile audience in pervasive environments. We introduced two agents who interact according to a learning by teaching model. A student agent is running on a mobile device and a teacher agent on remote. The common goal is to maximize the expected satisfaction of the user adapting applications and contents to be delivered and augmenting and improving his perception of the reality. Personal devices have been exploited to interact with the environment, to run interactive applications and to present contents to the user. We described a framework that implements the proposed approach. It represents preliminary research results about the aided exploitation and context awareness of complex archaeological sites by mobile devices. The framework supports the experts in the domain of the Cultural Heritage to augment the archaeological site with a set of multimedia contents which are delivered by innovative services to the visitors in order to guide their tour and to enhance their knowledge of the reality.
This work has been supported by PRIST 2009, Fruizione assistita e context aware di siti archelogici complessi mediante terminali mobile, founded by Second University of Naples.