Ubiquitous computing services started taking advantage of the reasoning capabilities of inference engines to acquire hidden and potentially useful contextual information. However, performance evaluations of the inference engines have been limited to the domain of static information reasoning; evaluations of requirements pertaining to ubiquitous computing environment have been largely neglected. This paper aims to examine how different types of inference engines perform by applying them to realistic ubiquitous computing scenarios. Based on the scenarios, three measurement criteria are proposed and measured including scalability as data set gets large, responsiveness for user's requests, and adaptability to frequent inference requests.
Ubiquitous computing services aim to provide information and services in more intelligent ways with more seamless interfaces that aid users to be conveniently served anytime, anywhere with any devices without awkward user intervention. These must rely on not only multiple sensors capturing user's context but also reasoning capabilities for processing raw sensed context data into more useful and meaningful information within the user's environment. Recently, inference engines such as Jena, RacerPro, FacT++, and Minerva have been proposed as a core component of intelligent ubiquitous computing systems. There had been many extensive evaluations of their reasoning capabilities in correctness, completeness, and response time [10,11]. It is less widely known without fully analyzing how well the inference engines can fare full-fledged ubiquitous computing services covering vast zones with various sets of requirements such as lots of fast moving users and other transient computing entities. The main purpose of this paper is to examine how well inference engines satisfy responsiveness requirement, scalability requirement and requirement to accommodate frequent inference requests in response to dynamic data insertion and deletion that realistic ubiquitous computing environments exhibit. To do so, we have modeled scenarios based in a major Korean university such as MyEntrance service. MyEntrance service scenario is a part of a larger Celadon project [8] in aim to study and adopt the most suitable reasoner for ubiquitous environment in its system. Specifically, five most prominent engines are considered based on their reasoning mechanisms: three memory-based and two DBMSbased engines.
MINERVA, JENA, Pellet, RacerPro, and DLDB-OWL (HAWK) [1,2,3,4,5,6,7] are discussed as representative instances of each class of reasoners and they are summarized in Table 1.
Minerva is high performance OWL storage, inference, and query system built on RDBMS. The advantages of the system could be categorized in two important aspects of a reasoner: response time and scalability. It does all required inferences in the time of data load instead of data query, making it more responsive at the time of user query. In addition, it calculates all inferences in relational database management system, making it more scalable than memory based counterpart. It is provided as a component of IBM's Integrated Ontology Development Toolkit (IODT) [4] and it supports DLP (Description Logic Program), a subset of OWL DL and conjunctive query, a subset of the SPARQL language. Minerva uses Description Logic reasoner for TBox and a set of logic rules translated from Description Logic Programming (DLP) for ABox inference. DLDB-OWL is a repository framework and toolkit that supports OWL It provides APIs as well as implementations for parsing, editing, manipulating and preservation of OWL ontologies. The architecture of DLDB-OWL consists of three packages: core, owl and storage. The core package defines the generic interfaces of the data structures of ontology and ontology objects, e.g. class and property. The core package, that is independent of underlying model the application will use, provides an API for constructing and manipulating ontology models. The owl package provides the utilities for parsing and serializing ontologies in OWL language.
Pellet is an open-source Java based OWL DL reasoner. It can be used in conjunction with both Jena and OWL API libraries and also provides a DIG interface. Pellet API provides functionalities to see the species validation, check consistency of ontologies, classify the taxonomy, check entailments and answer a subset of RDQL queries. It supports the full expressivity OWL DL including reasoning about nominals (enumerated classes).
RacerPro is an OWL reasoner and inference server for the semantic web. The origins of RacerPro are within the area of description logics. It can be used as a system for managing semantic web ontologies based on OWL. However, RacerPro can also be seen as a semantic web information repository with optimized retrieval engine because it can handle large sets of data descriptions.
Jena is a Java framework for building Semantic Web applications.
It provides a programmatic environment for RDF, RDFS and OWL, SPARQL and includes a rule-based inference engine. The Jena Framework includes a RDF API, reading and writing RDF in RDF/XML, N3 and N-Triples and an OWL API and SPARQL query engine.
For the performance evaluation on static information, we placed our focus on scalability and subsequent performance issue. Specifically, in evaluating the performance of query processing, we considered 16 sets of University Ontology Benchmark [9] queries that were generated by IBM China Research Lab. by extending widely used Lehigh University Benchmark [10].
In order to evaluate handling of context information, SPARQL is used as follows:
SELECT DISTINCT ?person ?zone ?Hobby1 ?Hobby2 ?Hobby3 WHERE (?person benchmark:locatedIn <http://rcubs.kyunghee.ac.kr/owl/rcubs-univ-benchdl.owl#Zone1>) (?person benchmark:locatedIn ?zone) (?person benchmark:like ?Hobby1)
For DLDB-OWL/HAWK evaluation, the above query is translated into KIF-like query as follows:
[http://rcubs.kyunghee.ac.kr/owl/univ-benchdl.owl] (type Person ?x) (locatedIn ?x http://rcubs.kyunghee.ac.kr/owl/rcubs-univ-benchdl.owl#Zone1) (locatedIn ?x ?z) (like ?x ?y1)
We set up our scenario environment of campus like the topology shown as Figure 1. For context generation, user's current location is gathered by the context event handler on the client's portable device or tags at any time the user is passing by sensors located in the entrance gate of the campus buildings. At a fixed cycle, the context event handler sends the user's current context information as OWL-DL format to the context server. The context server returns the result if the ontology files comes through successfully to the context event handler. The user may invoke the application server to get location-based service at any time the user wants.
Then the application server asks the context server to get servicerelated context data as facts. According to the facts, the inference engine determines the location-based services most appropriate for the user. We select query response time as the performance measure. The summary of response time is listed in Table 2. 51.50 Performance evaluations in case of dynamic context were also performed. Dynamic context evaluations were differentiated from the static context evaluation in that, engines were constantly requested to inference based on new data whereas the static evaluations were performed after the inference processing is finished. For this test, since the memory-based reasoning systems cannot be directly compared to the DBMS-based reasoning systems in their absolute loading time, we tested two databasebased reasoning systems: MINERVA and DLDB-OWL/HAWK. Only the test with data set 1 is described in this paper because we encountered a consistent problem with the dynamic data set and did not see the value in presenting more data in this study. The result is listed in
We first studied the responsiveness to the user request using both memory-based and DBMS-based inference engines in static context setting. The result indicated that database-based inference engines far outperform the other in this criterion owing to the fact that memory based inference engines pre-process context data while loading and thus is able to respond without further calculation at query time.
Then, using DBMS-based inference engines, we analyzed how they perform in conducting a context-aware MyEntrance service under the setting and environment of a major university. By varying the cycle of context changes and knowledge loading, performance measures such as correctness and completeness were examined. We noticed that engines do not respond to queries in the middle of their inferences and sometimes generate invalid or no responses.
We conclude that current state-of-the-art inference engines do not fulfill responsiveness, scalability and high-update frequency requirements demanded for ubiquitous computing environments with lots of fast moving users and other transient computing entities. And, evolutionary algorithms or inference mechanisms to deal with enormous amount of data, frequency of updates and respond to user's needs in time are needed for the inference engines to be improved.