This volume contains the papers presented at ORE 2015: The 4th OWL Reasoner Evaluation Workshop held on June 6, 2015 in Athens (co-located with the 28th International Workshop on Description Logics). The committee decided to accept 11 papers (6 Evaluation and Benchmarks papers and 5 System papers). Each submission was reviewed by at least 3 program committee members. OWL is a logic-based ontology language standard designed to promote interoperability, particularly in the context of the (Semantic) Web. The standard has encouraged the development of numerous OWL reasoning systems, and such systems are already key components of many applications. The goal of ORE, the OWL Reasoner Evaluation workshop, is to bring together both the developers and users of reasoners for (subsets of) OWL 2, including systems focusing on both intensional (ontology) and extensional (data) query answering. ORE 2015, as previous years, also included a competition in which OWL reasoners were faced with di erent reasoning tasks. The competition included six disciplines in which reasoners could compete: ontology classi cation, consistency checking, and realisation each for the OWL 2 EL and OWL 2 DL pro les. The tasks were performed on several large corpora of real-life OWL ontologies obtained from the web, as well as user-submitted ontologies which were found to be challenging for reasoners.
We thank all members of the program committee, competition organisers, additional reviewers, authors of the submitted papers, developers of the submitted reasoners and ontologies, and local organizers for their invaluable e ort.
We also thank Andreas Steigmiller (Ulm University) for providing us with the competition framework and Konstantin Korovin (University of Manchester) who kindly provided us with the PC cluster for the competition, supported by the Royal Society grant RG080491.
We also gratefully acknowledge the support of our sponsors: B2i Healthcare (https://www.b2international.com/) and DBOnto (http://dbonto.cs.ox. ac.uk/).
We would also like to acknowledge that the work of the ORE organisers was greatly simpli ed by using the EasyChair conference management system (http://www.easychair.org) and the CEUR Workshop Proceedings publication service (http://ceur-ws.org/).
Last but not least, we also thank Yevgeny Kazakov who brought the certi cates and betting slips, Despoina Trivela who dealt with the payment of the workshop T-shirts, and Bernardo Cuenca Grau and Yujiao Zhou who will most probably bring us a query answering track in ORE 2016 (this year they were almost there).
Birte Glimm Rafael Goncalves Matthew Horridge Ernesto Jimenez-Ruiz Nicolas Matentzoglu Bijan Parsia
Franz Baader Fernando Bobillo Claudia D'Amato Christine Golbreich Yevgeny Kazakov Evgeny Kharlamov Pavel Klinov Ilianna Kollia Markus Krotzsch Despoina Magka Francisco Martin-Recuerda Christian Meilicke Julian Alfredo Mendez Catia Pesquita Maria Del Mar Roldan-Garcia Joerg Schoen sch Floriano Scioscia Alessandro Solimando Weihong Song Kavitha Srinivas Andreas Steigmiller Valentina Tamma Dmitry Tsarkov Zhe Wang Dmitriy Zheleznyakov
University of Ulm Stanford University Stanford University University of Oxford University of Manchester University of Manchester
TU Dresden Universidad de Zaragoza University of Bari University of Montpellier The University of Ulm University of Oxford University of Ulm National Technical University of Athens TU Dresden Yahoo! Universidad Politecnica de Madrid University of Mannheim TU Dresden University of Lisbon, Faculty of Sciences Universidad de Malaga University of Mannheim Politecnico di Bari Universita di Genova University of New Brunswick IBM Research University of Ulm University of Liverpool University of Manchester Gri th University University of Oxford
In addition to workshop paper submissions, ORE 2015 also included a competition in which OWL reasoners were faced with di erent reasoning tasks. The competition included six disciplines in which reasoners could compete: ontology classi cation, consistency checking, and realisation each for OWL EL and OWL DL pro les. The tasks were performed on several large corpora of real-life OWL ontologies obtained from the web, as well as user-submitted ontologies which were found to be challenging for reasoners.
The ORE competition results are available here: http://dl.kr.org/ore2015/
TrOWL: http://trowl.org/ Konclude: http://www.derivo.de/en/produkte/konclude/ ELepHant: https://code.google.com/p/elephant-reasoner/ HermiT: http://www.hermit-reasoner.com/ MORe: http://code.google.com/p/more-reasoner/ ELK: http://code.google.com/p/elk-reasoner/ jcel: http://jcel.sourceforge.net/ FaCT++: http://code.google.com/p/factplusplus/ Jfact: http://sourceforge.net/projects/jfact/ Chainsaw: http://sourceforge.net/projects/chainsaw/ PAGOdA: http://www.cs.ox.ac.uk/isg/tools/PAGOdA/ Pellet: https://github.com/ignazio1977/pellet Racer: http://racer-systems.com/
We used the same competition framework as in 2014. It is available from GitHub https://github.com/andreas-steigmiller/ore-2014-competition-framework/. The ORE 2015 data set contains more than 16,500 unique ontologies. The set comprises: { the MOWLCorp (Manchester OWL Corpus), which was obtained through a Web Crawl, Google Custom Search API and user submissions (http:// mowlrepo.cs.manchester.ac.uk/datasets/mowlcorp/), { the Oxford Ontology Library (http://www.cs.ox.ac.uk/isg/ontologies/), { a BioPortal (https://bioportal.bioontology.org/) Snapshot (June 2014), { and user submitted ontologies such as BioKB, DMOP, FHKB, USDA, DPC, genomic-CDS, City-Bench, Cell ontology, DINTO, Virtual Fly Brain, GO, Drosophile Plenotype ontology.
iv
The ontologies in the data set are binned by pro les. For the competition, the EL pro le bin (8,805 ontologies) and the pure DL bin (7,704 DL ontologies that do not fall into one of the pro les) were used. Two further bins are obtained from these two bins by considering only the ontologies with an ABox (DL 2,439, EL 1,941 ontologies). The latter two are used for the realisation discipline, whereas the former ones are used for the classi cation and consistency checking disciplines.
Within these bins, the ontologies are further categorised by size (very small, small, medium, large, very large). A le list is then created by iterating over these categories (skipping categories that are already fully covered). From these le lists, the rst X are used for the competition, where X is chosen such that most reasoners are able to nish within a time limit (7 hours for classi cation and realisation, 3 hours for consistency checking). For classi cation X is 250 (OWL DL) and 300 (OWL EL), for consistency checking and realisation X is 200 (OWL DL) and 250 (OWL EL).
The whole data set is available for download at http://zenodo.org/record/ 10791 and more details about the corpus can be found at http://mowlrepo. cs.manchester.ac.uk/datasets/ore-2014/.
The competition was executed live on June 9th with a PC cluster at the University of Manchester provided by Konstantin Korovin. The machines of the cluster were equipped with an Intel Xeon QuadCore CPU running at 2.33GHz and 12GB RAM, where 10GB could be used by the reasoners. The reasoners were executed on the machines (one reasoner per machine) by running them natively on the used Fedora 12 operating system (64bit) or within a Java Runtime Environment (Java version 1.6). A three minute time limit was given every reasoner for each ontology, where 2.5 min was allowed for reasoning, i.e., 0.5 min could additionally/separately be used for parsing of the ontology and serialization of the result. Expected results were determined by a majority vote between the hash codes of the normalised results of those reasoners that terminated within the time limits. In case of a draw, one hash code was randomly chosen and declared as the expected hash code.