The OWL Reasoner Evaluation (ORE) Competition is an annual competition (with associated) workshop which pits OWL 2 compliant reasoners against each other on various standard reasoning tasks against corpora. The 2015 competition was the third of its sort and had 14 reasoners competing in 6 tracks comprising 3 tasks (consistency, classi cation, and realisation) over two pro les (OWL 2 DL and EL). In this paper, we discuss the design, execution and results of the 2015 competition with particular attention to lessons learned for benchmarking, comparative experiments, and future competitions.
The Web Ontology Language (OWL) is in its second iteration (OWL 2) and
has seen signi cant adoption especially in the Health Care and Life Sciences.
OWL 2 DL can be seen as a variant of the description logic SROIQ with the
various other pro les being either subsets (e.g., OWL 2 EL) or1 extensions (e.g.,
OWL 2 Full). Description logics generally are designed to be computationally
practical so that, even if they do not have tractable worst-case complexity for
key services, they nevertheless admit implementations which seem to work well
in practice [
Research on optimising OWL reasoning continues apace, though empirical work still lags theoretical and engineering work in breath, depth, and sophistication. There is, in general, a lack of shared understanding of test cases, test scenarios, infrastructure, or experiment design. A common strategy in research communities to help address these issues is to hold competitions, that is, experiments designed and hosted by third parties on an independent (often constrained, but sometimes expanded) infrastructure. Such competitions (in contrast to published benchmarks) typically do not directly provide strong empirical evidence 1 Some related standardised logics are subsets and extensions (e.g., RDFS) which are proper subsets of OWL 2 Full. about the competing tools. Instead, they serve two key functions: 1) they provide a clear, motivating event that helps drive tools development (e.g., for correctness or performance) and 2) components of the competition are useful for subsequent research. Finally, competitions can be great fun and help foster a strong community. They can be especially useful for newcomers by providing a simple way to gain some prima facie validation of their tools without the burden of designing and executing complex experiments themselves.
Toward these ends, we have been running a competition for OWL reasoners (with an associated workshop): The OWL Reasoner Evaluation (ORE) competition. ORE has been running, in substantively its current form, for three years and in this paper we describe the 2015 competition (held in conjunction with the 28th International Description Logic Workshop (DL2015)2 in June 2015). 2
The ORE competition is inspired by and modelled on the CADE ATP System Competition (CASC) [14, 22] which has been running for 25 years and been heavily in uential in the automated theorem proving community3 (esp. for rst order logic).
Key common elements: 1. A number of distinct tracks/divisions/disciplines characterised by problem type (e.g., \e ectively propositional" or \OWL 2 EL ontology"). 2. The test problems are derived from a large, neutral, updated yearly set of problems (e.g., for CASC, the TPTP library [21]). 3. Reasoners compete (primarily) on number of problems solved with a tight per problem timeout.
As description logics have a varied set of core inference services supported by essentially all reasoners, ORE also has track distinctions based on task (e.g., classi cation or realisation). Other CASC inspired elements: 1. ORE 2015 consisted entirely of a \live" competition run during the DL workshop. 2. There was a secondary competition among DL attendees to predict the results for various reasoners. 3. Competitors and organisers were given custom designed t-shirts.
We observe that central to such competitions is participation, thus various incentives to participate are critical especially in the early years of the competition as it is trying to get established. Hence the importance of \fun" elements, incentives (e.g., prizes, bragging rights), as well as a reasonable chance of winning at least something. 2 The websites for DL2015 and ORE2015 are archived at http://dl.kr.org/dl2015/ and https://www.w3.org/community/owled/ore-2015-workshop respectively. 3 See the CASC website for details on past competitions: http://www.cs.miami.edu/ ~tptp/CASC/. Also of interest, though not directly inspirational for ORE, is the SAT competition http://www.satcompetition.org// 2.1
ORE 2015 had 6 tracks based on three central reasoning services (consistency, classi cation, and realisation) and two OWL pro les (OWL 2 DL and EL). Classi cation is, almost certainly, the most common and important reasoning service for ontologies to date. Consistency is, in some sense, the most fundamental service. Realisation gets us at least a minimal form of instance reasoning. These services are not ubiquitously supported, with realisation not handled by some reasoners. These have standard de nitions (though any consequence equivalent de nition would do): { An ontology O is consistent if O 6j= > v ? and inconsistent otherwise. { The classi cation of an ontology O (Cl(O)) is f j = A v B; A; B 2 Nc [ f?; >g; O j= g where Nc is the set of class names in O. { The realisation of an ontology O (Rl(O)) is f j = A(x); x 2 Ni; A 2 Nc; O j= g where Nc is the set of class names, Ni is the set of individual names in O.
We split out a track into a tractable pro le when we have enough participants which are speci cally tuned for that pro le. In prior years we have had an RL and QL track, but the number of RL and QL speci c reasoners is very low. We believe this is, in part, due to the fact that RL and QL users tend to be conjunctive query oriented. We hope to introduce a conjunctive query track in future years, but see the discussion below for some of the challenges there. All reasoners purporting to handle the entirety of OWL 2 DL are entered in all tracks. Thus we have specialised EL reasoners competing against complete OWL DL reasoners.
For each track, we award prizes to the top three participants for a total of 18 possible winners. 2.2
The full competition corpus contains 1,920 ontologies, sampled from three source
corpora: A January 2015 snapshot of Bioportal [12] containing 330 biomedical
ontologies, the Oxford Ontology Library4 with 793 ontologies that were collected
for the purpose of ontology related tool evaluation and MOWLCorp [7], a
corpus based on a 2014 snapshot of a Web-Crawl containing around 21K unique
ontologies. Each competition comes with its own random strati ed sample of
ontologies from this base corpus - this means that not all 1,920 ontologies actually
made it into the live competition. Ontology processing was done using the OWL
API (3.5.1) [
As a rst step, the ontologies of all three source corpora were collected and serialised into OWL/XML with their imports closure merged into a single ontology. The merging is, from a competition perspective, necessary to mitigate the bottleneck of loading potentially large imports repeatedly over the network and 4 http://www.cs.ox.ac.uk/isg/ontologies/ because the hosts of frequently imported ontologies sometimes impose restrictions on the number of simultaneous accesses.5 After the collection, the entire pool of ontologies is divided into three groups: (1) Ontologies with less than 50 axioms, (2) OWL 2 DL ontologies, (3) OWL 2 Full ontologies. The rst group is removed from the pool. As reasoner developers may chose to tune their reasoners towards the ontologies in the three publicly available source corpora, we included a number of approximations into our pool. The entire set of OWL 2 Full ontologies was approximated into OWL 2 DL, i.e., we used a (slightly modi ed) version of the OWL API Pro le checker to drop enough axioms so that the remainder is in OWL 2 DL. As some degree of OWL Fullness comes from illegal axiom interaction,6 we repeated the \DLi cation" process twice. The OWL DL group was then approximated into OWL 2 EL and OWL 2 QL, using the approximation method employed by TrOWL [15]. As the only syntax that is uniformly supported by all reasoners participating the competition, we then serialised the current pool (including the original OWL 2 DL ontologies, the EL/QL-approximated ontologies and the \DLi ed" OWL 2 Full ontologies) into Functional Syntax, and gathered all relevant ontology metrics again. As some ontologies are included in more than one of the source corpora, we excluded at this point (as a last preprocessing step) all duplicates from the entire pool of ontologies and removed ontologies with TBoxes containing less than 50 axioms. This left us with the full competition dataset of 1,920 unique OWL 2 DL ontologies. The random strati ed sampling for the competition then was done as follows: All ontologies were binned by size into the following groups: Very small (50-99 axioms), small (100-999 axioms), medium (1,000-9,999 axioms), large (10,000-100,000 axioms) and very large (more than 100,000 axioms). From each group, we attempted to sample 60 original ontologies, and 15 approximated ones for each competition. For the OWL 2 EL related track, the ontologies had to fall under the OWL 2 EL pro le, for the OWL 2 DL competition the ontologies had to fall under OWL 2 DL but not under any of the three OWL 2 pro les, and for the two realisation challenges we only considered those ontologies that had at least 100 ABox axioms. This process resulted in the following six live competition corpora: 109 for OWL 2 EL realisation, 298 for OWL 2 EL classi cation and consistency, 264 for DL realisation and 306 for DL consistency and classi cation.
Figures 1 and 2 show the ontology sizes in terms of axiom counts and the usage of constructs through the corpus.
The full competition corpus, and the execution order of the competition, can be obtained from Zenodo [9]. 5 Which may be exceeded considering that all reasoners in the competition run in parallel. 6 For example, an added declaration might introduce an illegal punning. clas ifcation instantiation 1000
100000 clas ifcation 1000 TBoxsize 100000 consistency 1000
100000 instantiation 10 1000 100000 The test framework used in ORE 2015 is a slightly modi ed version of the one used for ORE 2014 which is open sourced under the LGPL and available on Github.7
The framework takes a \script wrapper" approach to running reasoners instead of, for example, requiring all reasoners to use (a speci c version of) the OWL API. While this puts some extra burden on established reasoners with good OWL API bindings this, combined with the requirement only to handle some OWL 2 standard syntax (with the very easy to parse and serialise Functional Syntax [11] as a fairly common choice), makes it very easy for new reasoners to participate even if they are written in hard-to-integrate with the JVM languages. The OWL API also is a very rich and rather heavyweight framework that is not tightly integrated with most reasoners. For example, systems using the OWL API generally consume more memory because they maintain the OWL API level representation of the ontology and the reasoner internal one. Thus, avoiding the OWL API can help competition performance. However, there is a standard script for OWL API based reasoners so it is fairly trivial to prepare an OWL API wrapped reasoner for competition. 7 https://github.com/andreas-steigmiller/ore-2014-competition-framework/.
A detailed description of the framework and how to run it is available there.
However, this is not necessarily a desirable outcome as encouraging reasoners to provide good OWL API support (thus supporting access to those reasoners by the plethora of tools which use the OWL API) is an outcome we want to encourage.
Reasoners report times, results, and any errors through the invocation script.
Times are in wall clock time (CPU time is inappropriate because it will penalise
parallel reasoners) and exclude \standard" parsing and loading of problems (i.e.,
without signi cant processing of the ontology). The framework enforces (con
gurable) timeouts for each reasoning problem. Results are validated by comparison
between competitors with a majority vote/random tie breaking fallback strategy.
Note, unlike CASC, we do not require reasoners to produce proofs of their
results as this is not a standard feature of description logic reasoners and for many
services (such as classi cation) it may be impractical. We are however
experimenting with a more satisfactory justi cation-based technique for disagreement
resolution [
The framework supports both serial and parallel execution of a competition. Parallel distributed mode is used for the live competition but serial mode is su cient for testing or o ine experiments. The framework also logs su cient information to allow \replaying" the competition and includes scripts for a complete replay as well as jumping to the nal results.
The competition was run on a cluster of 19 machines: 1 master machine that dispatched reasoners with problems to the 18 client machine as well as collecting and serving up results to a live display. Each machine sported an Intel Xeon 4-Core L5410 running at 2.33GHz with 12GB of RAM, for which 2GB were reserved for the operating system (i.e., 10GB could be used by the reasoners). The operating system was Ubuntu 14.04.02 LTS and the Java version was OpenJDK v1.7.0 64-bit. The reasoner execution was limited to 180s for each ontology in each track, where only 150s were allowed for reasoning and 30s could additionally be used for parsing and writing results in order to reduce the penalisation of reasoners with slow parsers. Hence, only if the time reported by the reasoner exceeded 150s was it interpreted as a timeout. 3
There were 14 reasoners participating with 11 purporting to cover OWL 2 DL and 3 being OWL EL speci c, see Table 1. There is no speci c penalty or test for being incomplete with respect to a pro le and, indeed, one reasoner, TrOWL is intentionally incomplete for performance reasons.
The number of participants is fairly stable over the past three years ranging
from 11 to 14. There is a stable core of participants with some uctuation on the
margin. Some reasoners are not entered by their original developers (e.g., Pellet)
and ORE currently has no policy against that. We anticipate in the future that
more coalition reasoners will be made available, though currently only MORe
and Chainsaw use component reasoners (ELK and HermiT the former, FaCT++
the latter) which are also competing. MORe's coalition involves partitioning the
ontology into an EL and DL part, dispatching each part to the respective tuned
reasoner, and combining the results [
Given the presence of deliberately incomplete (with respect to their purported pro le) reasoners we are considering whether to modify the voting procedure to discount those reasoners' votes in certain cases. A full break-down of all tracks and competing reasoners can be seen in Table 2. Results, error reports and more details on the competition framework are available at http://dl.kr.org/ore2015. Figure 3 shows the results of all partici8 HermiT was submitted with OWL API 3 and OWL API 4 bindings pants in all tracks as displayed during the live competition. During the competition, these charts are dynamically updated as problems are being solved and reported.
Out of the 6 competitions, 4 were won by the new hybrid reasoner
Konclude [20], and two (EL-consistency and EL-classi cation) were won by ELK [
MOReHermiT r elephant elk konclude MOReHermiT TrOWL
Ontologies
This is particularly obvious for the EL-classi cation competition. Up until a certain point, Konclude is doing much (sometime up to an order of magntitude) better than ELK (the winner of the discipline), but towards the harder end, ELK overtakes Konclude. Some of this may be due to JVM overhead for ELK and our \ re and forget" execution strategy. If we had a long running server based approach it might be that the JVM overhead for easy cases would be e ectively amortised. Another interesting observation is the performance of ELepHants [18] consistency check, which regularly outperforms both ELK and Konclude. We speculate that this is due to di erences in whether parsing time is incorporated in the reported time (e.g., ELK does this for all tasks and Konclude does this for consistency checking). ) c se10.0 ( e m it g n i n o s a e
R0.1
A full break-down for all reasoners by competition can be seen in Table 3.
The competition is reasonably challenging: In only two tracks (EL consistency and EL classi cation) did any reasoner solve all the problems in competition conditions. Figure 6 shows a detailed breakdown of how many problems were solved by how many reasoners (in percent).
Ontologies Reasoner Task Success Timeout Error 4" refers to the current version of HermiT wrapped in the OWL API version 4.
It is interesting to observe that the union of all reasoners successfully process all EL reasoning problems. As one might expect, realisation is still challenging for reasoners. But in all tracks, for the majority of reasoners, the ORE problems provide a good target for optimisation. We know, from the results of the competition, that these problems are (almost) all in principle solvable on a modest machine in around 3 minutes. 5
The top slots in all tracks have been dominated by Konclude (and to a lesser extend by ELK) for two years now. Konclude is an highly optimised, very e cient reasoner whose developers continuously test it against a vast set of available ontologies. Even so, there is interesting jockeying around second and third place 200 150 100 50 5 11 26 13 8 17 27 18 38 51 tn 0 u o c200 150 100 67 50 0 1 3 15 9 5 36 14 25 0 0 30 60 92 123 for all tracks, and we were impressed with how well older reasoners, which have not been updated recently (notably Pellet and Racer), fared.
The robustness experiments in [
However, the ORE workshop solicits \challenge" ontologies from ontology developers partly in the hopes of directing reasoner developer attention to real user performance needs. Unfortunately, we have not yet managed to do a \user ontology" track, though we are hoping to do so as a satellite event at OWLED 2015. This will almost certainly have to be o ine and, of course, many of the submitted ontologies are currently unsolved by current reasoners.
The most important next expansion of tracks is to conjunctive query answering (CQA). Setting up a meaningful CQA competition is signi cantly more di cult, because we do not only have to consider ontologies, but also queries and data. Gathering suitable (meaningful) queries is probably the most di cult hurdle to overcome. However, we made signi cant progress toward a reasonable design this year and hope to incorporate it in next year's competition.
Another area of interest is application style benchmarks which would situate the reasoning task in the context of a pattern of use characteristic of a real or realistic application. This might include modi cation of the ontology or data during the competition run.
The ORE 2015 Reasoner Competition continues the success of its predecessors. Participants, workshop attendees, and interested bystanders all had fun, and the ORE 2015 corpus, whether used with the ORE framework or in a custom test harness, is a signi cant and distinct corpus for reasoner experimentation. Developers can easily rerun this years competition with new or updated reasoners to get a sense of their relative progress and we believe that solving all the problems in that corpus in similar or somewhat relaxed time constraints is a reliable indicator of a very high quality implementation.
Acknowledgments. The ORE competition has been the work of many people over the years and we would like to especially acknowledge the contributions of Ernesto Jimenez-Ruiz for running the very rst primitive competition and being a PC chair for all workshops; Samantha Bail for being a PC chair as well as implementing the rst \live results" screen; Ian Horrocks for helping getting the project started; and Yevgeny Kasakov for helpful discussions on the competition design as well as nding a critical bug just before the competition started. We also would like to acknowledge the generous support of B2i Healthcare9 for their repeated donations of prize money and the DBOnto project10 for funding compeition T-Shirts.
Finally, the competition would not have been possible without the donation of cluster time by Konstantin Korovin (funded by Royal Society research grant RG080491). 9 http://b2i.sg 10 http://www.cs.ox.ac.uk/projects/DBOnto/