Context-aware systems, such as in-car infotainment systems, aim at improving the interaction between computers and humans by using contextual information about the system, the user and the environment. Previous work has shown that ontologies have signi cant bene ts for modeling such context-aware systems, e.g., when inferring knowledge. However, the potential performance impact on the overall system when adopting ontologies, especially with rules, to model context-awareness is yet unknown. In this paper, we introduce a benchmark infrastructure and perform benchmarks on multiple ontologies of context-aware systems in order to determine factors that in uence performance. From the results of these benchmarks, we derive guidelines for designing ontologies with rules for context-aware systems. These guidelines allow making conscious decisions about performance trade-o s and, in consequence, may improve suitability of ontologies for use in implementing industrial context-aware systems as they guide the creation of high-performance ontology-based context-aware systems.
Context-aware systems aim at improving the interaction between computers and
humans by using contextual information about the system, the user and their
environment to provide suitable functionality for a particular context. For
example, context-aware in-car infotainment systems may adapt to the respective
drivers, their current situation and their intentions. In such a system, the car
may, e.g., play rock music when the driver was identi ed as being tired. To
realize such change in functionality, contextual values have to be captured and their
e ect on the system has to be speci ed in suitable models. In previous work [
This paper is structured as follows: In Section 2, we provide background information on context-aware systems and their modeling. In Section 3, we introduce the benchmark setup used to inspect the performance of ontology-based context-aware in-car infotainment systems. In Section 4, we describe the execution and present the results of our benchmarks by revealing signi cant in uences certain properties of ontologies have on the performance of the overall system. In Section 5, we discuss these results and use them to derive design guidelines for building high-performance ontology-based context-aware systems. In Section 6, we elaborate on related work. Finally, in Section 7, we summarize the contributions and provide an outlook to future work to conclude the paper. 2
Context-aware systems consider contextual values to alter their functionality to
be suitable for a speci c operator in a speci c situation and a particular
environment. Context-aware systems receive low level contextual information (e.g.,
from sensors), process these to high level contextual information and react to the
change in context by altering their functionality appropriately [
Context-aware systems from the automotive domain, especially in in-car
infotainment systems, distinguish three main categories of contextual information:
the driver, the car and their environment [
Despite the bene ts of modeling context-aware systems with ontologies [
In this section, we introduce the software setup that allows benchmarking the performance of ontology-based context-aware systems. Furthermore, we elaborate on the procedure to generate random ontologies to diversify the input to the benchmark in order to determine the performance of ontology-based contextaware systems. 3.1
As foundation of the benchmark, we employ a component based architecture.
The respective components are used to represent the individual functions found
in a context-aware system as well as to synthesize data creation and
consumption in the form of mock-ups. As we are using ontologies to understand the
current situation of the user as described in [
Source A
Context Data
Source B
Context Data
Source C generates Context Receiver Context Supplier
Context Management
Context Receiver Context Supplier
Context Receiver Context Supplier
Ontology as .owl le Context Data
Supplier I
Context Data
Supplier II
Context Data Supplier III generates overview of the software architecture used for executing benchmarks based on such software components.
Context Data Sources provide contextual information in a generic format for the Context Management. The Context Management loads the model in terms of an ontology saved in the OWL le format and creates individuals within the provided ontology for each contextual information sent by any Context Data Source and updates these individuals whenever it receives new data. Hence, the number of individuals will never exceed the number of classes within the ontology. Afterwards, the Context Management starts reasoning on the ontology to infer knowledge, e.g., the Context Management could reason about the driver's condition based on his or her interaction with the car. For this purpose, the benchmark infrastructure uses the Pellet Reasoner. In a last step, the Context Management queries the ontology for added inferred knowledge and noti es the corresponding Context Data Supplier of any changes.
As described in [
When the Context Management launches, it creates generic Context Receivers that establish connections to every Context Data Source. In addition, names of Context Data Suppliers are attached to every Output Class de ned within the ontology. Hence, the Context Management can again create generic Context Suppliers that establish connections to their corresponding Context Data Supplier during run-time. 3.2
It is our intent to benchmark a wide variety of ontologies for context-aware
systems following di erent design approaches. Furthermore, to reliably determine
the impact of di erent design approaches on performance, a signi cant size of
an ontology is required, which exceeds the size of the case studies used in our
previous work [
For these reasons, we decided to devise a generation algorithm for ontologies following di erent design approaches that have may be varied in size. Furthermore, we also created a procedure to generate appropriate software components that are used to interact with the ontology during run-time, e.g., Context Data Sources and Context Data Supplier. Our ontology generation algorithm is aligned with best practices of ontologies that were created by domain experts of ontology-based context-aware in-car infotainment systems during several industrial projects.
Ontology Generation To ensure that the results measured from our benchmark are not speci c to a particular design approach for ontologies, we decided to generate several ontologies following di erent design approaches. For example, we align the depth of the inheritance hierarchy with our experience from industrial practice but allow parametrization of the number of direct subclasses. These are automatically generated with respect to a set of manually de ned parameters: { the amount of Input and Output Classes { the amount of SWRL rules used for reasoning Using these parameters, we are able to create a wide variety of ontologies that can be used as input to the benchmark as well as to generate Context Data Sources and Context Data Supplier. We assured that the generated ontologies closely resemble our industrial modeling practices for ontologies in a contextaware in-car infotainment.
Code Generation The basic implementation of the benchmark architecture presented in Figure 1 is provided as generic components, such as the Context Management, which can be reused in various scenarios. To further benchmark concrete context-aware systems based on ontologies, we generate Context Data Sources and Context Data Supplier from a speci ed ontology. During run-time, the generated Context Data Sources re data change events for each of their Input Classes in arbitrary intervals to ensure a constant system load. Context Data Suppliers receive noti cations whenever one of their Output Classes was inferred by the Context Management. Hence, the model and every software component of our benchmark are either generic or automatically generated by a corresponding code/ontology generator.
Using the previously described setup, we performed benchmarks for various different ontologies. This section elaborates on the method of execution for the benchmarks and presents the respective results regarding the performance of using ontologies to model context-aware systems. 4.1
We make two assumptions how an ontology's size and the amount of SWRL rules used for reasoning may in uence the performance of the overall system: Assumption 1: An ontology's size has a negative e ect on the performance of the overall system. Bigger ontologies are expected to slow down the system performance.
Assumption 2: The number of SWRL rules used for reasoning has a negative impact on the performance of the overall system. Ontologies with more SWRL rules are exected to slow down the system performance. However, to renounce on SWRL rules would massively limit expressiveness of our model of contextual information.
To check these assumptions, we create two metrics representing two di erent time frames during run-time of an ontology-based context-aware system: { Reasoning Time: The time it takes the Pellet Reasoner to do the reasoning on the ontology during run-time. { Query Time: The time that is needed to query the ontology for inferred knowledge and to notify a Context Data Supplier. 4.2
To check our rst assumption, we generated multiple ontologies of di erent sizes (i.e., 10, 50, 100, 150 and 200 classes). For each size, we generated three ontologies. Every one of these was stressed in three separate runs by receiving data from three di erent Context Data Sources. We stopped the recording as soon as the reasoning had made 1; 000 cycles.
Figure 2a shows that the reasoning time of ontologies increases exponentially with an increasing size of the ontology. Even more important, the standard deviation of the reasoning time increases dramatically with size. The maximum reasoning time that occurred during our benchmark of ontologies with 200 input classes was as high as 2; 128:0ms and the minimum was as low as 1:0ms by a mean of 16:74ms. The standard deviation was at 29:56ms. This implies, that the reasoning time of larger ontologies cannot be predicted as accurately as that of smaller ontologies.
Figure 2b shows a signi cant increase of mean query time as well as its standard deviation. This means that it takes much more time to query larger (c) Mean reasoning time and standard deviation compared to ontologies with and without SWRL rules. ontologies for inferred knowledge than it takes for smaller ontologies. In addition, the accuracy of predictions regarding the required query time deteriorates with the size of an ontology as can be seen by the high standard deviation. For ontologies with 200 classes, we measured query times between 20ms and 12; 765ms at a mean of 2472ms and a standard deviation of 1514ms. To check our second assumption, we compared the execution time of the previously used ontologies with that of similar ontologies without rules. For this purpose, we removed all SWRL rules from all three ontologies of size 100 and again performed 1; 000 reasoning cycles in three separate runs.
Figure 2c compares the results of ontologies with and without SWRL rules. The results show that the existence of SWRL rules does not have any signi cant e ect on the reasoning time of an ontology. The mean reasoning time is almost una ected (4:283ms with SWRL rules compared to 4:279ms without SWRL rules) and standard deviation increases only slightly when using SWRL rules (7:636ms with SWRL rules compared to 7:044ms without SWRL rules). We also recorded a higher maximum value when reasoning ontologies with SWRL rules (531ms with SWRL rules compared to 369ms without SWRL rules). 5
In this section, we discuss the implications of the results of our benchmarks presented in Section 4. Furthermore, we use the insights gained from these results to derive guidelines for the modeling of high-performance ontology-based context-aware in-car infotainment systems.
For one, the benchmark results show that the reasoning time increases signi cantly with the number of classes in the ontology, which may lead to a bad performance of the overall system as the reasoner blocks computational resources during reasoning. Hence, our rst assumption can be con rmed. Furthermore, the results also show that SWRL rules within ontologies do not have any signi cant impact on the overall system performance as mean and standard deviation are very close with and without SWRL rules. Hence, our second assumption cannot be con rmed. This provides liberty to modelers creating an ontology-based context-aware in-car infotainment system with regard to using SWRL rules in their ontologies and, hence, build even more powerful ontologies.
Derived from the performed benchmarks and presented results, we de ne the following design guidelines for modeling high-performance ontology-based context-aware in-car infotainment systems.
As our benchmark results have shown, an increasing number of classes within
an ontology has a negative impact on the performance of an ontology-based
context-aware in-car infotainment system. In absolute terms, we were pleasantly
surprised that the Pellet Reasoner performed su ciently well even with larger
ontologies. However, at the current time, the overall approach for modeling
incar infotainment systems based on ontologies presented in [
If the context-aware in-car infotainment system depends on a large amount of contextual information that is the basis for reasoning, an ontology and, there exist distinct groups of contextual information without any connections between them, it is well advised to avoid modeling a single monolithic ontology. To improve the performance of reasoning, the ontology should be split up over multiple constituent ontologies that contain elements with particularly high cohesion but only reference the ontologies that contain elements with less cohesion (in respect to the rst ontology). Especially on a multi-core hardware, the overall system performance would bene t, as the Pellet Reasoner currently does not to seem to automatically optimize reasoning for systems with more than one CPU core. For context-aware in-car systems, a suitable structuring of ontologies might be aligned with the aforementioned distinction of contextual information regarding the driver, the car and the environment.
Our benchmark results show that SWRL rules only have a marginal impact on
the performance of an ontology-based context-aware in-car infotainment system.
In consequence, this means that the approach presented in [
Following these guidelines, the modeler of a context-aware in-car infotainment system may capitalize on ontologies with rules for a both powerful and high-performance system. 6
Several di erent types of context models can be found in the literature. An
overview is provided in the survey paper by Strang and Linnho -Popien [
In this paper, we introduced a benchmark infrastructure to inspect the performance of modeling context-aware systems using ontologies. We presented a generator for plausible ontologies and associated source code, which both respect design principles derived from industrial practice. We performed multiple benchmarks on various di erent ontologies and derived guidelines for the design of ontology-based context-aware systems. With these guidelines, we intend to foster the adoption of ontologies by providing means for modeling and implementing high performance context-aware systems based on ontologies, such as in-car infotainment systems.
In our future work, we will optimize querying ontologies for inferred knowledge, as this seems to be the major issue when aiming for a high-performance ontology-based context-aware system. In addition, we will investigate other properties of ontologies that potentially have an impact on the overall system performance, such as the depth of the inheritance hierarchy in the ontology. Finally, we will evaluate the accuracy of the benchmark results within an industrial setting.