A Performance Evaluation of OWL 2 DL Reasoners using ORE 2015 and Very Large Bio Ontologies

Abstract

In this paper, we evaluate the reasoning performance of six prominent OWL 2 DL reasoners, using two collections of ontologies: ORE 2015 and the 21 largest ontologies from the NCBO BioPortal. We observed that the majority of the reasoners were unable to successfully perform over half of the reasoning tasks in the NCBO BioPortal dataset which includes some very large ontologies. Despite of being a representative selection of the state-of-the-art OWL 2 DL reasoners, it came to our attention that many of them are no longer being actively maintained. These findings serve as a cautionary message to the OWL 2 reasoning community, emphasizing the need to focus research eforts towards keeping up with the growing requirements posed by new and very large ontologies and knowledge graphs.

Keywords

• eol>OWL 2
• Ontology
• OWL 2 Reasoners
• Performance Evaluation

1. Introduction

are graph databases that support these two recommendations. The evaluation presented in [ 7 ] demonstrated that state-of-the-art triplestores can handle very large knowledge graphs such as Wikidata. However, most of these triplestores do not support OWL 2 DL, or they only support one of the tractable fragments, such as OWL 2 RL. Apache Fuseki2 is an example of the former case, and RDFox3 is an example of the latter case.

Because many triplestores do not support OWL 2 DL or only one of its fragments, it may not be possible to guarantee the correctness of queries if the knowledge graph includes certain OWL 2 constructors. This could be the case with the knowledge graph YAGO 4 [ 3 ], which defines the 6 top classes as disjoint classes, meaning they cannot share any individuals. A careless extension of YAGO 4 may easily violate these design restrictions, and only an OWL 2 DL reasoner might be able to identify these violations. Therefore, it might be recommended to adopt an OWL 2 DL reasoner when working with OWL 2 DL knowledge graphs. However, as many of the available reasoners were designed to test diferent reasoning algorithms, optimizations, and extensions beyond OWL 2 DL, deciding on an appropriate reasoner for a particular application can be challenging.

In this paper, we evaluate the performance of six prominent OWL 2 DL reasoners, namely Pellet [ 8 ], FaCT++ [ 9 ], JFact4, Openllet5, HermiT [ 10 ], and Konclude [ 11 ]. Except for Konclude, these reasoners support the OWLAPI library6, which is one of the most relevant ontology and semantic knowledge graph development frameworks. To test these reasoners, we selected two collections of OWL 2 ontologies: ORE 2015 [ 12 ] and the 21 largest ontologies from the NCBO BioPortal [ 13 ]. The evaluation scripts and results of this work can be found at the GitHub repository7.

The remainder of the paper is structured as follows. Section 2 discusses related work. Section 3 describes the evaluation setup. Section 4 provides a detailed discussion of the evaluation results. Finally, Section 5 concludes the paper and presents future work directions.

2. Related Work

During the past two decades, a number of benchmarks and frameworks have been introduced to support the comparison and evaluation of the reasoners. Some of these benchmarks were based on generating synthetic datasets such as [ 14, 15, 16, 17, 18 ]. These early initiatives were used to test several representative reasoners at that time on a relatively low number of ontologies and a small set of hand-crafted queries [ 19 ]. The LUBM benchmark [ 20 ], together with its extensions - UOBM [ 21 ] and SLUBM [ 22 ], are the most representative and popular synthetic benchmarks with varying sizes of instances for the university ontology. However, due to several shortcomings in their design (e.g., sparsely interrelated data and very simple schemas), these cannot be considered as meaningful benchmarks as many reasoners that achieved excellent results in these benchmarks failed other ABox or TBox tests [ 23 ]. In contrast, using real-world 2https://jena.apache.org/documentation/fuseki2/ 3https://www.oxfordsemantic.tech/product 4https://github.com/owlcs/jfact 5https://github.com/Galigator/openllet 6https://github.com/owlcs/owlapi 7https://github.com/SINTEF-9012/owl-reasoner-evaluation ontologies to evaluate the performance of the reasoners was a diferent direction where the results can be easily interpreted and exploited [ 24 ]. In this light, several benchmarks and evaluations were introduced in [ 24, 25, 26, 27 ]. However, these works used just a small set of ontologies and thus cannot be used to test the scalability of the reasoner. The ORE benchmark which was a part of OWL Reasoner Evaluation (ORE) Competition [ 12 ] contains a large dataset of ontologies of varying sizes sampled from diferent ontology libraries. This benchmark can be used to evaluate the reasoners in terms of the coverage of OWL 2 constructors and scalability.

This work is motivated by the ORE competition in order to provide a thorough understanding of state-of-the-art OWL 2 DL reasoners to assist developers in the selection of appropriate solutions for their semantic applications. We focus only on reasoning performances as there were several comprehensive surveys and comparisons of reasoners with regard to other aspects such as query language, programming interface and OWL 2 constructors coverage as in [ 18, 28, 29 ]. In this light, a new evaluation of state-of-the-art reasoners with the stress on scalability and large-scale knowledge graph is in need. There are several new releases of the reasoners since the last ORE 2015 competition, while the most recent benchmark papers, such as OWL2Bench (2020) [30] and the evaluation (2022) presented in [ 19 ], did not include all well-known reasoners or focused mainly on energy profiling. Due to these shortcomings, we decided to conduct a new performance evaluation using the most recent versions of the reasoners with the ORE 2015 dataset. In addition, in order to investigate how these reasoners can perform on very large and modern ontologies, we selected the 21 largest ones from the NCBO BioPortal.

3. Evaluation Setup

The evaluation was conducted on Amazon Elastic Compute Cloud (EC2) r5.2xlarge8 instances equipped with 8 vCPUs of Intel Xeon Platinum 8000 series processor at up to 3.1 GHz, 64 GB RAM, Ubuntu 18.04 LTS, and 100 GB EBS gp3 volume which ofers SSD-performance with predictable, baseline performance of 3,000 IOPS and 125 MB/s throughput9.

3.1. Reasoners

We aim to evaluate the performance of state-of-the-art reasoners in order to support the selection of appropriate libraries for semantic data processing and reasoning applications. In this regard, we selected only OWL 2 DL reasoners that are still actively maintained within the last 10 years. The reasoners should provide interfaces for OWLAPI (either version 4 or 5) - the ontology development framework supported by most reasoners [29] - in order to enable the integration into our benchmark programs. Based on those criteria, the following six reasoners: Pellet, FaCT++, JFact, Openllet, HermiT, and Konclude are selected.

Table 1 lists the latest releases (before March 2023) of the evaluated reasoners. Except for Pellet and FaCT++ which support OWLAPI 4, the others provide interfaces for the latest OWLAPI 5. Furthermore, as Konclude only provides a standalone version with support for OWLLink protocol10, the OWLLink-OWLAPI library was used to integrate the reasoner into OWLAPI. 8https://aws.amazon.com/ec2/instance-types/r5/ 9https://aws.amazon.com/ebs/general-purpose/ 10https://www.w3.org/Submission/owllink-structural-specification/

3.2. Datasets

The evaluation has been carried out on two datasets: ORE 2015 [ 12 ] and NCBO BioPortal [ 13 ] ontologies. The ontologies are binned by size into the following groups: very small (less than 1K axioms), small (1K - 10K axioms), medium (10K - 100K axioms), large (100K - 1M axioms), very large (1M - 10M axioms), and huge (more than 10M axioms), as illustrated in Figure 1.

(a) ORE 2015 Ontologies (Loading, Consistency, Classification) 624 (b) ORE 2015 Ontologies (Realization) (c) NCBO Bio-ontologies s 466 488 e i g o lt o n O f o r e b m u N 317

25

Very SmalSlmall MediumLargeVery LargeHuge

112 225 145 141 1

Very SmalSlmall MediumLargeVery LargeHuge

Size of Ontologies 1 4 1

Very SmalSlmall MediumLargeVery LargeHuge

15

ORE 2015 contains 1920 ontologies from the OWL Reasoner Evaluation 2015 Competition dataset11, which can be used to evaluate the performance of the reasoners on small and mediumsize ontologies. Regarding the reasoning task realization, we only consider ontologies with more than 100 ABox axioms. This process results in a subset of 624 ontologies for the realization task.

NCBO BioPortal is one of the largest libraries of biomedical ontologies and terminologies. Ontologies have been selected by taking the 20 largest ones in this portal. Additionally, the GALEN ontology [31] was also added to this dataset as it is known to be one of the most dificult ontologies for tableau reasoners. This ontology contains many cyclic axioms, which result in very large models and an extremely long time to classify and realize [32]. Table 2 shows the statistics of those 21 ontologies. This dataset is used to evaluate the performance of the reasoners on very large real-world ontologies and knowledge graphs. NCBO Bio-ontologies statistics: number of Classes, Ind[ividuals], Axioms, Logical Axioms, TBox, ABox, Data Prop[erties], and Obj[ect] Prop[erties]. The list were ordered by total number of axioms.

3.3. Experiment Configuration

To run the experiment, we implemented two small Java programs using OWLAPI to: (1) perform a reasoning task T on an ontology O using a particular reasoner R, and (2) run the evaluation with the sets of reasoners Rs, ontologies Os, and tasks Ts. By splitting the evaluation into two diferent programs, each reasoning task for a reasoner on an ontology can be run in a separate process. Particularly, for each evaluation iteration, the ontology is loaded again, and a new instance of the reasoner for the corresponding ontology is created. Therefore, this setup eliminates the efect of memory limitation and caching on the measured execution times. The default settings were applied for all reasoners as we assume that the general users do not have too much knowledge to optimize the configurations. There are four diferent tasks were evaluated: • Loading reasoner: regards the time for initializing a new instance of the reasoner and loading the ontology model into it. • Consistency: verifies whether every class in the ontology admits at least one individual. • Classification : computes a class hierarchy with all superclasses and subclasses of every • Realization: for each individual, it finds all classes, especially the most specific ones, class defined in an ontology.

where the individual is an instance of.

The timeout was set to 30 minutes for ORE 2015 dataset and 1 hour for BioPortal ontologies. Our primary performance measure is the task execution time. Each reasoning task was evaluated for 10 iterations, and the mean values were used for the analysis. We propose to use both arithmetic mean and geometric mean (ℎ root of the product over numbers) of the execution time. To compute the means, the failed tasks (e.g., timeout, errors) are penalized with double the timeout value (1 hour for ORE 2015 and 2 hours for BioPortal ontologies). Arithmetic mean is dominated by large numbers, so it can be used as an indicator of a high ratio of success and failure tasks (i.e., smaller value indicates more success tasks). Geometric mean can handle varying proportions and mitigate the efect of large number penalty. Therefore, geometric mean is used to evaluate the overall performance over success tasks (i.e., smaller value indicates smaller execution time for the success tasks).

4. Results and Discussions

4.1. ORE 2015 Dataset task reasoning times are plotted in Figure 2, where timeouts and errors were excluded. Regarding loading task, JFact ranked first with no error or timeout, while Openllet and

HermiT

JFact

Konclude

Pellet

Openllet g n i d a o L ) a ( 100

1 0.01 0.0001

100 y c n e its 1 s n o C ) (b 0.01 0.0001

100 n o i t a iifc 1 s s a l C )c 0.01 ( 0.0001

100 n o i t iza 1 l a e R ) (d 0.01 0.0001

Very Small (<1K)

Small(<10K)

Medium(<100K) Large &

Very Large (<10M)

Konclude are in the top three with up to 3 timeouts. Although Openllet was developed based on Pellet, the reasoner had much better results with no errors, while Pellet had the most errors. Konclude had one error related to OWLLINK library (i.e., not supported DatatypeDefinition). HermiT, FaCT++, and Pellet did not succeed in loading all ontologies due to a lot of errors. Most of the errors are related to OWL constructs (e.g., SWRL rule uses a built-in atom, unsupported datatype). Regarding the execution time, JFact and FaCT++ seem to load ontologies faster than the others as they have the smallest geometric mean, while Konclude amounted to the largest value of geometric mean. As OWLLink library also validates consistency after loading ontologies, Konclude took a longer time to complete this initialization and loading task while having much better results in the consistency validation task. This insight can also be noticed in Figure 2(a), where Konclude usually lies above others while FaCT++ and JFact are the lowest.

Regarding the consistency validation task, Konclude and HermiT were in the top two, with the least timeouts and errors. Although JFact was loaded fastest, it had the most timeouts in this task. JFact also had the largest geometric mean, indicating that the reasoners had the poorest performance on this task. Furthermore, Pellet ranks last due to many errors related to unsupported datatype (17% of the ontologies), while its descendant, Openllet, is in the fourth position with only 4 errors. Regarding the overall performance of the reasoners on success ontologies, Konclude has the best performance with the smallest geometric mean. As seen from Figure 2(b), Konclude has approximately constant performance on this task because consistency checking was already done when loading the reasoner, as discussed earlier. On the other hand, HermiT can perform even better than Konclude on many small ontologies. Furthermore, it is worth mentioning that although Openllet and Pellet vary a lot in the number of errors, these two reasoners have very similar performance on all reasoning tasks on success ontologies. This can be seen in Figure 2(b), where Openllet lies just slightly above Pellet in all four plots. This ifnding shows that the Openllet reasoner was developed to resolve the limitation of Pellet on OWL features support without significant improvement in reasoning performance.

Konclude and HermiT were also in the top two reasoners for classification and realization tasks, while JFact and Pellet always ranked last when comparing the number of success ontologies. Furthermore, although FaCT++ had more failures than Konclude and HermiT, it performs better on small ontologies. This can be seen from the geometric mean as well as in 2(c) and (d), where FaCT++ had the best performance on small ontologies. Openllet and Pellet are still in the middle range, while HermiT performed slower than these two reasoners, although HermiT had much fewer timeouts. JFact still had the poorest performance.

In summary, Konclude and HermiT were always in the top two reasoners with the most successful reasoning tasks on the ORE 2015 ontologies, although they did not always have the best reasoning times. Specifically, Konclude is more scalable as the reasoner had the best performance on large and very large ontologies. HermiT had very good performance on consistency validation. However, for classification and realization tasks, this reasoner was outperformed by the other reasoners. Therefore, it is most suitable for applications that require only consistency validation. Furthermore, FaCT++ was the next reasoner in the list with more timeouts and errors than HermiT but had much better performance on small ontologies. This reasoner can be considered for applications with small knowledge graphs or ontologies with simple OWL constructs. Pellet and its descendant Openllet were always in the middle range with slight slower performance than FaCT++. Furthermore, Openllet was developed based on Pellet with substantial improvements related to OWL syntax support. However, regarding task execution performance, Openllet and Pellet had very similar results. Finally, JFact consistently ranked last in the evaluation.

4.2. NCBO BioPortal Dataset

evaluate the performance of the reasoners on large-scale knowledge graphs, we selected the 21 largest ontologies. The results for the reasoner loading task are consistent with those of ORE 2015, where JFact was the fastest reasoner and Konclude (Kc) ranked last due to additional time required for consistency validation. Konclude also had the highest number of failures, which were attributed to memory overflow. Upon closer examination of these OOM errors, we found that they were related to the OWLLINK library, which is not designed to handle very large ontologies. However, when we used KoncludeCLI - the standalone version of Konclude - to perform the reasoning tasks on the same ontologies, we did not encounter any errors. Additionally, the table shows that Openllet performed slightly better than its previous version Pellet - and was also capable of handling several syntax errors that occurred with Pellet.

Konclude performed exceptionally well in the consistency task, owing to its ability to validate consistency during the reasoner loading process. However, when we compared the loading time of Konclude with the consistency validation time of the other reasoners, HermiT emerged as the top-performing reasoner in this test. This outcome aligns with the results of the ORE 2015 evaluation, which also found HermiT to be superior in checking ontology consistency, particularly with very large ontologies.

In terms of classification and realization, Konclude outperformed the other reasoners overall. JFact had the poorest performance, failing on all ontologies in this evaluation. Additionally, HermiT had fewer errors than FaCT++, Openllet, and Pellet. However, when considering only the successful cases, HermiT appeared to run slower than Openllet and Pellet, a finding consistent with the results of ORE 2015. Finally, FaCT++ exhibited inferior performance on these two reasoning tasks, suggesting it may not be eficient for very large ontologies.

Overall, the results of the NCBO bio-ontologies evaluation were consistent with the findings of the ORE 2015 evaluation. Konclude emerged as the top-performing reasoner, while JFact failed on nearly all ontologies. Notably, Konclude was the only reasoner to successfully complete all tasks on the GALEN ontology, which is considered one of the most dificult ontologies due to its many cyclic axioms. However, Konclude may encounter memory overflow errors when performing reasoning tasks on some very large ontologies, due to limitations with OWLLINK library. This issue does not arise when using KoncludeCLI. HermiT was the second-best performing reasoner with fewer errors and decent execution results, followed by Openllet, Pellet, and FaCT++, all of which had failures on almost half of the ontologies.

5. Conclusion and Future Work

This paper presents a comprehensive evaluation of the state-of-the-art OWL 2 DL reasoners using two real-world ontology datasets from ORE 2015 and NCBO BioPortal. The evaluation results from both datasets were consistent, with Konclude and HermiT frequently ranking at the top for successful reasoning tasks, while JFact had the most failures. In terms of reasoning time, Konclude demonstrated superior performance, while the others showed varying performance across diferent reasoning tasks and ontology sizes. Our evaluation identifies the strengths and weaknesses of each reasoner, providing valuable insights for ontology developers and application designers when selecting the most suitable reasoning system for their specific needs.

During the evaluation of the reasoners, it came to our attention that many of them are no longer being actively maintained. This is apparent from Table 1, which shows that several reasoners have not been updated in the past five years. Additionally, the majority of reasoners were unable to successfully perform over half of the reasoning tasks in the NCBO BioPortal dataset, which consists of large to very large ontologies. These findings serve as a cautionary message to the OWL 2 reasoning community, emphasizing the need to focus research eforts towards keeping up with the ever-evolving complexity of the ontology language and the growing requirements of semantic applications.

As for future work, we aim to expand this evaluation by incorporating large and prominent OWL 2 knowledge graphs such as DBpedia, Wikidata or YAGO. These knowledge graphs have gained significant attention across various domains and have been adopted in many applications.

Acknowledgments

The authors would like to thank the anonymous reviewers for their valuable feedback. This work has been funded by The Research Council of Norway projects SkyTrack (No 309714), DataBench Norway (No 310134) and SIRIUS Centre (No 237898), and the European Commission projects DataBench (No 780966), VesselAI (No 957237), Iliad (No 101037643), enRichMyData (No 101070284), Circular TwAIn (No 101058585), and Graph-Massivizer (No 101093202). [29] N. Matentzoglu, J. Leo, V. Hudhra, U. Sattler, B. Parsia, A Survey of Current, Stand-alone

OWL Reasoners., in: ORE, Citeseer, 2015, pp. 68–79. [30] G. Singh, S. Bhatia, R. Mutharaju, OWL2Bench: A Benchmark for OWL 2 Reasoners, in:

International Semantic Web Conference, Springer, 2020, pp. 81–96. [31] A. Rector, J. Rogers, Ontological and Practical Issues in Using a Description Logic to Represent Medical Concept Systems: Experience from GALEN, in: Reasoning Web International Summer School, Springer, 2006, pp. 197–231. [32] Y. Kazakov, M. Krötzsch, F. Simančík, The Incredible ELK, Journal of Automated Reasoning 53 (2014) 1–61.

References

[1] D. Vrandečić , M. Krötzsch , Wikidata: A Free Collaborative Knowledgebase , Communications of the ACM 57 ( 2014 ) 78 - 85 .

[2] J. Lehmann , R. Isele , M. Jakob , A. Jentzsch , D. Kontokostas , P. N. Mendes , S. Hellmann , M. Morsey , P. Van Kleef , S. Auer , et al., DBpedia - A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia , Semantic web 6 ( 2015 ) 167 - 195 .

[3] T. Pellissier Tanon , G. Weikum , F. Suchanek , YAGO 4: A Reason-able Knowledge Base , in: The Semantic Web: 17th International Conference, ESWC 2020 , Heraklion, Crete, Greece, May 31-June 4, 2020 , Proceedings 17, Springer, 2020 , pp. 583 - 596 .

[4] B. Motik , P. F. Patel-Schneider , B. Parsia , C. Bock , A. Fokoue , P. Haase , R. Hoekstra , I. Horrocks , A. Ruttenberg , U. Sattler , et al., OWL 2 Web Ontology Language Structural Specification and Functional-Style Syntax , W3C recommendation ( 2009 ).

[5] D. W. Richard Cyganiak , M. Lanthaler, RDF 1.1 Concepts and Abstract Syntax, W3C recommendation ( 2014 ).

[6] S. Harris , A. Seaborne , SPARQL 1 . 1 Query Language, W3C recommendation ( 2014 ).

[7] A. N. Lam , B. Elvesaeter , F. Martin-Recuerda , Evaluation of a Representative Selection of SPARQL Query Engines using Wikidata, in: The Semantic Web - 20th International Conference, ESWC 2023, Hersonissos, Crete, Greece, May 28 - June 1, 2023 , Proceedings, Springer, 2023 .

[8] E. Sirin , B. Parsia , B. C. Grau , A. Kalyanpur , Y. Katz , Pellet: A practical OWL-DL reasoner , Journal of Web Semantics 5 ( 2007 ) 51 - 53 .

[9] D. Tsarkov , I. Horrocks , FaCT++ Description Logic Reasoner: System Description , in: International Joint Conference on Automated Reasoning , Springer, 2006 , pp. 292 - 297 .

[10] B. Glimm , I. Horrocks , B. Motik , G. Stoilos , Z. Wang , HermiT: An OWL 2 Reasoner, Journal of Automated Reasoning 53 ( 2014 ) 245 - 269 .

[11] A. Steigmiller , T. Liebig , B. Glimm , Konclude: System description , Journal of Web Semantics 27 ( 2014 ) 78 - 85 .

[12] B. Parsia , N. Matentzoglu , R. S. Gonçalves , B. Glimm , A. Steigmiller , The OWL Reasoner Evaluation (ORE) 2015 Competition Report , Journal of Automated Reasoning 59 ( 2017 ) 455 - 482 .

[13] N. F. Noy , N. H. Shah , P. L. Whetzel , B. Dai , M. Dorf , N. Grifith , C. Jonquet , D. L. Rubin , M.-A. Storey , C. G. Chute , et al., Bioportal: ontologies and integrated data resources at the click of a mouse , Nucleic Acids Research 37 ( 2009 ) W170 - W173 .

[14] P. F. Patel-Schneider , R. Sebastiani , A New General Method to Generate Random Modal Formulae for Testing Decision Procedures , Journal of Artificial Intelligence Research 18 ( 2003 ) 351 - 389 .

[15] J. Hladik , A Generator for Description Logic Formulas , in: Description Logics, 2005 .

[16] F. Massacci , F. M. Donini , Design and Results of TANCS-2000 Non-Classical ( Modal) Systems Comparison , in: Automated Reasoning with Analytic Tableaux and Related Methods: International Conference, TABLEAUX 2000 , St Andrews , Scotland, UK , July 3- 7 , 2000 Proceedings 9 , Springer, 2000 , pp. 52 - 56 .

[17] I. Horrocks , P. F. Patel-Schneider , DL Systems Comparison , in : Proc. of the 1998 Description Logic Workshop (DL'98) , volume 11 , 1998 , pp. 55 - 57 .

[18] J. Bock , P. Haase , Q. Ji , R. Volz , Benchmarking OWL Reasoners, in: ARea2008-Workshop on Advancing Reasoning on the Web: Scalability and Commonsense , Tenerife Spain, 2008 .

[19] F. Scioscia , I. Bilenchi, M. Ruta , F. Gramegna , D. Loconte , A multiplatform energy-aware OWL reasoner benchmarking framework , Journal of Web Semantics 72 ( 2022 ) 100694 .

[20] Y. Guo , Z. Pan , J. Heflin, LUBM: A benchmark for OWL knowledge base systems , Journal of Web Semantics 3 ( 2005 ) 158 - 182 .

[21] L. Ma , Y. Yang , Z. Qiu , G. Xie, Y. Pan , S. Liu, Towards A Complete OWL Ontology Benchmark , in: The Semantic Web: Research and Applications: 3rd European Semantic Web Conference , ESWC 2006 Budva, Montenegro, June 11-14, 2006 Proceedings 3, Springer, 2006 , pp. 125 - 139 .

[22] T. N. Nguyen , W. Siberski, SLUBM: An Extended LUBM Benchmark for Stream Reasoning ., in: OrdRing@ ISWC, 2013 , pp. 43 - 54 .

[23] T. Weithöner , T. Liebig , M. Luther , S. Böhm , F. Von Henke , O. Noppens , Real-World Reasoning with OWL , in: The Semantic Web: Research and Applications: 4th European Semantic Web Conference, ESWC 2007 , Innsbruck, Austria, June 3-7, 2007 . Proceedings 4, Springer, 2007 , pp. 296 - 310 .

[24] Z. Pan , Benchmarking DL Reasoners Using Realistic Ontologies ., in: OWLED , volume 188 , 2005 .

[25] C. Tempich , R. Volz , Towards a benchmark for Semantic Web reasoners - an analysis of the DAML ontology library , in: EON , volume 87 , 2003 .

[26] T. Gardiner , D. Tsarkov , I. Horrocks , Framework For an Automated Comparison of Description Logic Reasoners, in: The Semantic Web-ISWC 2006 : 5th International Semantic Web Conference, ISWC 2006 , Athens, GA, USA, November 5- 9 , 2006 . Proceedings 5, Springer, 2006 , pp. 654 - 667 .

[27] K. Dentler , R. Cornet , A. Ten Teije , N. De Keizer, Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile, Semantic Web 2 ( 2011 ) 71 - 87 .

[28] S. Abburu , A survey on ontology reasoners and comparison , International Journal of Computer Applications 57 ( 2012 ).