<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>First Results on How to Certify Subsumptions Computed by the E L Reasoner Elk Using the Logical Framework with Side Conditions?</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Franz Baader</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Patrick Koopmann</string-name>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Cesare Tinelli</string-name>
          <email>cesare-tinelli@uiowa.edu</email>
          <xref ref-type="aff" rid="aff0">0</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>Department of Computer Science, The University of Iowa</institution>
          ,
          <country country="US">USA</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Theoretical Computer Science</institution>
          ,
          <addr-line>TU Dresden</addr-line>
          ,
          <country country="DE">Germany</country>
        </aff>
      </contrib-group>
      <abstract>
        <p>The generation of proof certi cates and the use of proof checkers is nowadays standard in rst-order automated theorem proving and related areas. They have, to the best of our knowledge, not yet been employed in Description Logics, where the focus was on detecting and repairing errors in the ontology, rather than on catching erroneous consequences created by an incorrect reasoner. This paper reports on rst steps towards remedying this de cit for subsumptions computed by the DL reasoner Elk. We use an existing tool for generating proofs of consequences from Elk, and transform these proofs into a format that is accepted as certi cates by our proof checker. The checker is obtained as an instance of a generic certi cation tool based on the Logical Framework with Side Conditions (LFSC), by formalizing the inference rules of Elk in LFSC. We report on the results of applying this approach to the classi cation of a large number of real-world OWL 2 EL ontologies.</p>
      </abstract>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>-</title>
      <p>The purpose of this paper is to show that tools developed in rst-order
automated theorem proving (ATP) and satis ability modulo theories (SMT) for
certifying reasoning results can in principle also be employed in Description
Logic (DL) to increase the trust in reasoning results (such as the subsumption
hierarchy) computed by DL reasoners.</p>
      <p>Highly-optimized automated reasoning tools are complex software systems,
and thus may produce erroneous results due to programming errors, even if
soundness and completeness of the underlying calculus have been proved in
detail. If the results of the reasoning process are used in safety-critical situations
(e.g., when verifying software), then it is important that one can trust these
results. Since it is currently not possible to verify a large and sophisticated
software system like an automated theorem prover, the solution to this problem is
? Partly funded by the DFG grant 389793660 as part of TRR 248.</p>
      <p>
        Copyright c 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
that the prover outputs a proof certi cate for the result, which can then easily
be checked using a proof checker. In contrast to the provers, proof checkers are
rather simple pieces of software, and thus are easier to trust or verify. For this
reason, the generation of proof certi cates is now common in (general-purpose)
ATP [
        <xref ref-type="bibr" rid="ref29 ref33">33,29</xref>
        ] and more specialized related areas such as SMT [
        <xref ref-type="bibr" rid="ref31 ref8">31,8</xref>
        ]. For example,
in most divisions of the ATP system competition CASC [
        <xref ref-type="bibr" rid="ref32">32</xref>
        ], it is required that
the participating systems output proofs for theorems and ( nite) counter-models
for non-theorems, though this output is not checked automatically. Proofs are
used in particular when ATP and SMT tools are integrated in other reasoning
tools such as skeptical proof assistants [
        <xref ref-type="bibr" rid="ref2 ref9">2,9</xref>
        ] or certifying software veri ers [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ].
      </p>
      <p>
        Since most DLs [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] are decidable fragments of rst-order logic, dedicated
decision procedures for speci c DLs are usually more e cient for reasoning on DL
knowledge bases than general-purpose theorem provers. Though DL reasoning
is simpler than ATP, e cient DL reasoners1 employ sophisticated optimizations
and data structures, and are thus again complex software systems that may
contain programming errors. In addition, while the correctness of the basic
calculi (such as tableaux or consequence-based calculi) may have been proved in
detail, this is not always the case for optimized variants. Nevertheless, the DL
community has mainly concentrated on explaining [
        <xref ref-type="bibr" rid="ref15">15</xref>
        ] and repairing [
        <xref ref-type="bibr" rid="ref16 ref7">16,7</xref>
        ]
errors in the input (i.e., the ontology that is classi ed), rather than on catching
errors created by the reasoning process. In the 2015 OWL Reasoner Evaluation,
the results produced by the reasoners were \validated by comparison between
competitors using a majority vote/random tie-breaking fallback strategy" [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ],
which in some cases unfairly penalized a correct reasoner [
        <xref ref-type="bibr" rid="ref20">20</xref>
        ].
      </p>
      <p>
        This paper reports on rst steps towards remedying this de ciency, where
we mainly tried to employ existing software rather than implementing new one.
As reasoning task we consider classi cation for OWL 2 EL ontologies,2 i.e., the
computation of the subsumption hierarchy between the concepts de ned in the
given ontology. Our aim is to certify subsumption relationships that have been
computed by the consequence-based DL reasoner Elk [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ]. For this purpose, we
use the inference tracing approach of [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] extending Elk to extract proofs of
computed subsumptions, and then transform these proofs into a format that is
accepted as certi cates by our proof checker. This checker is obtained by
instantiating a generic checker based on the Logical Framework with Side Conditions
(LFSC) [
        <xref ref-type="bibr" rid="ref31">31</xref>
        ] with an encoding in LFSC of a su ciently large subset of Elk's
inference rules. LFSC is a meta-framework based on the established Edinburgh
LF framework [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], which combines ease of presenting proof systems with a
high-performance checker for proofs in the represented systems.
      </p>
      <p>
        We have evaluated our approach on a large number of real-world OWL 2 EL
ontologies, obtaining promising results. Our experiments show that the certi
cates are usually of manageable size and can be checked in reasonable time,
1 See, e.g., [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ] for a list of the reasoners that participated in the OWL Reasoner
      </p>
      <p>Evaluation (ORE) in 2015.
2 OWL 2 EL is a pro le of the standard Web Ontology Language OWL 2 https://www.
w3.org/TR/owl2-pro les/.
though creating them may take a relatively long time. The latter problem could
be mitigated by developing reasoners that directly generate certi cates, rather
than extracting them from outputs of existing tools not built for this purpose.
2</p>
    </sec>
    <sec id="sec-2">
      <title>Certi cate Generation and Proof Checking</title>
      <p>Certi cate generation in ATP and SMT takes various forms depending on the
query issued to the prover and its result. When the query consists in showing that
a goal formula ' is entailed by a set of hypotheses and the prover succeeds in
proving that, the generated certi cate typically consists in a term that encodes a
proof of ' from or, equivalently, a proof of the unsatis ability of [ f:'g. In
the DL context, ' could be a subsumption statement C v D and an ontology.
If the subsumption holds, then consequence-based calculi provide us with a proof
of ' from whereas tableau-based calculi yield a proof of the unsatis ability of
with an a ABox of the form fC(a); :D(a)g, which describes a counterexample
to the subsumption statement.</p>
      <p>
        The encoding of proofs varies depending on the proof system in which the
proof is expressed and the granularity of proof terms. Some provers, such as
Vampire [
        <xref ref-type="bibr" rid="ref28">28</xref>
        ] and Z3 [
        <xref ref-type="bibr" rid="ref23">23</xref>
        ], generate proof terms that are in fact proof sketches:
an external checker needs proof search capabilities in order to reconstruct a full
proof from the sketch. Others, such as veriT [
        <xref ref-type="bibr" rid="ref11">11</xref>
        ], provide ne-grained proof terms
that can be checked with no proof search but require the checker to provide native
support for certain data structures (such as sets, clauses, or sequents) used in the
proof term. Finally, other provers, such as CVC4 [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ], provide ne-grained proofs
as terms in a logical framework expressive enough to formalize also the proof
system the proof is based on (such as LF [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], ELF [
        <xref ref-type="bibr" rid="ref27">27</xref>
        ] or the calculus [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]).
The latter require a proof checker that can take as input both the proof term
to be checked and its proof system. Logical frameworks are typically based on
a dependently-typed higher-order logic. This provides not only representational
power but also the ability to uniformly recast proof checking as type checking. A
prover's proof system is modeled as a type system T , and a proof term represents
a correct proof exactly when it is well typed in T .
      </p>
      <p>
        The last approach to proof checking provides the highest level of exibility
because the same proof checker can be used for very di erent provers and proof
systems as long as those systems are representable in the underlying logical
framework. The approach also provides a high level of trust in principle, for two
reasons. First, a generic checker that has been used successfully with many proof
systems is arguably more trustworthy than one custom-made for a speci c proof
system, unless the latter is very simple. Second, because the proof system is an
input to the proof checker, it is expressed as set of proof rules whose soundness
can be proved separately with a proof assistant, such as Coq [
        <xref ref-type="bibr" rid="ref34">34</xref>
        ] or Lean [
        <xref ref-type="bibr" rid="ref24">24</xref>
        ],
also based on a dependently-typed higher-order logic. This e ectively removes
the rules from the trusted core, which then reduces to just the proof checker.
::= type j
x: :
(Types)
::= int j k j t j
      </p>
      <p>
        x: 1[fp tg]: 2
(Terms) t ::= x j c j t: j x[: ]: t j t1 t2
For this work, we have used the LFSC [
        <xref ref-type="bibr" rid="ref31">31</xref>
        ], a logical framework that extends LF,
the Edinburgh Logical Framework [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ], with a bare-bones functional
programming language to express procedural side conditions for proof rules. Intuitively,
the extension to side conditions allows one to de ne proof systems more
compactly, with proof rules that delegate low-level checks on the rule's premises
(such as, for instance, that a given term occurs in a given premise) to a side
program. We refer the interested reader to Stump et al. [
        <xref ref-type="bibr" rid="ref31">31</xref>
        ] for more details on
the use of side conditions and focus on the main language instead.
      </p>
      <p>
        A slightly simpli ed version of LFSC's abstract syntax is provided in Figure 1.
It is an extension of LF's calculus, itself an extension of the simply typed
-calculus. The calculus has three levels of semantic entities, all denoted
by terms: values; types, understood as collections of values; and kinds, families
of types. The constant type denotes the kind of types. Types and kinds can be
dependent on (i.e., indexed by) values. Syntactically, if 2[x] is a type term whose
set of free term variables is fxg and 1 is a type term with no free variables, the
expression x: 1: 2[x] denotes in the calculus the (dependent) type of functions
that return a value of type 2[v] for each value v of type 1 for input x. When
2 has no free variables, the type x: 1: 2 is just the type 1 ! 2 of simply
typed -calculus,3 and we will use the latter notation for it, treating ! as right
associative. The same sort of parametrization applies to kinds as well, allowing
one for instance to de ne the type of vectors of size n with a type constant vec
of kind n:int: type where int is the prede ned type of mathematical integers.
LFSC adds to the calculus the possibility of imposing restrictions on the
parameters of a depended type. These restrictions are expressed operationally
by side conditions of the form fp tg where p is a program in the side condition
language and t is an LF term. The restriction, enforced at type checking time,
is that p does not fail and its result is equal to (more precisely, matches) t.
The LFSC Checker
The LFSC checker developed at the University of Iowa is a small, high
performance type checker for LFSC written in C++.4 It takes as input one or more
3 The type of unary functions with inputs of type 1 and outputs of type 2.
4 The checker is available in source form at https://github.com/CVC4/LFSC.
signature les containing an LFSC encoding of a proof system with side
conditions, and a le containing the certi cate, the proof term to be checked, and
reports whether the term is a correct proof or not. For any of the LFSC checker's
applications, the trusted core consists of the signature(s) used and the checker
itself which has, however, a rather small code base. Although the checker was
developed originally to check SMT proofs [
        <xref ref-type="bibr" rid="ref31">31</xref>
        ], and is used mainly for that
purpose,5 thanks to its generality it has also been used for other kinds of reasoners,
such as SAT solvers [
        <xref ref-type="bibr" rid="ref25">25</xref>
        ], model checkers [
        <xref ref-type="bibr" rid="ref22">22</xref>
        ] and, in our case, DL reasoners.
In fact, we argue that the LFSC checker (or similar logical framework-based
checkers such as Dedukti [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]) is particularly well-suited for Description Logics
given the large number of such logics and proof systems for them. In this work,
we chose the proof system adopted by the high-performance reasoner Elk for
the logic E L [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] mainly because Elk produces proof traces for its subsumption
checks, which can be converted to full LFSC proofs. The next section explains
how we did that and how we formalized ELK's proof system in LFSC.
3
      </p>
    </sec>
    <sec id="sec-3">
      <title>Certifying OWL EL Classi cation Results</title>
      <p>
        + [
        <xref ref-type="bibr" rid="ref4 ref6">4,6</xref>
        ] since it is
We focus on the subset of OWL 2 EL corresponding to E L?
supported by Elk and is expressive enough to provide us with a large number
of real-world ontologies for our experiments. Starting with mutually disjoint sets
+ concepts are constructed
NC and NR of concept and role names, respectively, E L?
from concept names using the constructors top concept (&gt;), bottom concept (?),
+ ontologies are then
conjunction (C u D), and existential restriction (9r:C). E L?
      </p>
      <p>
        nite sets of axioms of the form C v D for concepts C; D (concept inclusion,
CI) and r1 : : : rn v r for role names r1; : : : ; rn; r (role inclusion, RI). The
+ concepts and ontologies, which de nes how interpretations I
semantics of E L?
assign sets CI to concepts C and under what conditions an interpretation is a
model of an ontology, is formalized in the usual way (see [
        <xref ref-type="bibr" rid="ref4 ref6">4,6</xref>
        ] for details).
+
One of the most important inference problems is subsumption: given E L?
+ ontology O, we say that C is subsumed by D w.r.t.
concepts C; D and an E L?
O (written C vO D) if CI DI holds for all models I of O. The subsumption
+ is known to be decidable in polynomial time [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ]. DL systems
problem in E L?
o er classi cation as basic inference service, i.e., when reading in an ontology
O, they usually compute all subsumption relationships between the concept
names occurring in O, as well as &gt; and ?. Some DL systems actually compute
(and store) not the full set of these subsumptions, but their transitive reduct,
by leaving out those relationships that can be obtained by transitivity from
others. Whereas systems that use tableaux-based subsumption reasoning realize
classi cation by repeated calls of a basic subsumption algorithm [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ],
consequencebased approaches [
        <xref ref-type="bibr" rid="ref17 ref4">4,17</xref>
        ] classify the whole ontology in one go.
5 The SMT solvers CVC3 and CVC4 produce LFSC proofs.
R0
      </p>
      <p>C v C
R9 C v C9rv:D9r:DE v E</p>
      <p>R
&gt; C v &gt;</p>
      <p>R</p>
      <p>C v D u E
u C v D C v E</p>
      <p>Ru+ C Cv vD DCu vE E
R? C v 9Cr:Dv ?</p>
      <p>D v ?</p>
      <p>
        Rv CC vv DE : D v E 2 O
R C0 v 9r1:C1 C1 v 9r2:C2 : : : Cn 1 v 9rn:Cn : r1 : : : rn v r 2 O
C0 v 9r:Cn
For E L+?, Elk follows a consequence-based approach for classi cation [
        <xref ref-type="bibr" rid="ref19">19</xref>
        ], which
basically starts with the CIs in O, and then uses classi cation rules (see
Figure 2 for example rules) to add derived subsumptions. This saturation process
terminates after a polynomial number of rule applications, and a subsumption
between concept names in O follows from O i it is contained in the saturated
set computed by Elk. Note that, for e ciency reasons, in some of the rules the
axioms of the ontology O are distinguished from the derived subsumptions by
being used as side conditions rather than as premises.
      </p>
      <p>The OWL 2 EL standard de nes a range of axioms that can be seen as
syn+-axioms. In addition to the rules shown here, Elk uses rules
tactic sugar for E L?
that convert those axioms into CIs or RIs, and some other rules we do not discuss
for space constraints. To allow the certi cation of Elk proofs, we designed and
implemented an LFSC signature specifying datatypes and rules corresponding
to the language constructs and rules used in Elk. In the source code of Elk, we
found 50 classes implementing the interface ElkInference for rule applications.
+-axiom,
We restricted ourselves to the inferences on (syntactic variations of) E L?
and identi ed the rules actually used in the corpora discussed in the next section.
This resulted in a set of 18 rules which we then implemented in LFSC.
fP v R u S; P v 9r:T; T v Ug</p>
      <p>O</p>
      <p>P v P
P v 9r:T</p>
      <p>P v 9r:U</p>
      <p>R0</p>
      <p>Rv</p>
      <p>TT vv UT RRR09v</p>
      <p>P v P
P v R u S</p>
      <p>P v R R+Ru1</p>
      <p>u
R0</p>
      <p>Rv
P v 9r:U u R
(a) Elk proof</p>
      <p>P; R; S; T; U : conc</p>
      <p>r : role
o = P v R u S ; P v 9 r T ; T v U ; ;
p = R0 (P v P)
check (Ru+
(R9
(Rv
(Rv
(Ru1
) : h(P v 9 r U u R)
o p)
o (R0 (T v T))))
(Rv o p))
(b) LFSC proof
For illustration purposes, Figure 3 contains a fragment of the LFSC signature,
in abstract syntax, showing how the rst seven Elk rules from Figure 2 could
be encoded in LFSC.6 The rst three rows declare a number of types (with conc
for concepts, ont for ontologies, and so on) and constants corresponding to the
+
symbols of E L?. For increased readability, for constants we use the same symbols
+
as the corresponding operator in E L?. The subset of ontology axioms used in
the proof are encoded as a sequence of axiom constructed with the operators ;
and ;. The type constant h is used to construct proof judgments, statements of
the form h(a) expressing that axiom a is provable.</p>
      <p>A proof rule is represented by a dependently-typed constant whose type
directly encodes the rule's premises and conclusion. We name each such constant as
its corresponding rule in Figure 2, except for rules Ru1 and Ru2 which correspond
respectively to the rst and the second conclusion of rule Ru from that gure.
For brevity, we use notation like x; y: 1: 2 as a shorthand for x: 1: y: 1: 2.
The type of each rule is parametrized by term variables that correspond to the
schema variables in the Elk rule. For instance, R+, which has type
u
c:conc: d:conc: e:conc: h(c v d) ! h(c v e) ! h(c v d u e);
is parametrized by variables c, d, and e of type conc which correspond,
respectively, to the schema variables C, D, and E of the corresponding Elk rule. The
return type h(c v d u e) of Ru+ encodes the conclusion C v D u E of the Elk
rule whereas the argument types h(c v d) and h(c v e) encode the two premises
C v D and C v E. The other rules are similar. Note that the type of Rv
encodes the side condition of the corresponding Elk rule by means of an LFSC
side condition that checks whether the axiom c v d occurs in the input ontology
o. This is done by the side condition program in (whose de nition is not shown),
which scans o looking for c v d and returns tt if, and only if, it nds it.
6 The actual LFSC signature for the Elk calculus, including the side condition code,
is provided in full in the appendix in the Lisp-like concrete syntax of LFSC.
with the optimization that the two occurrences of subterm (R0 (P v P)) are
factored out by the de ned constant p. In each rule application, underscores
are used for arguments whose value does not need to be speci ed as it can be
generated by type inference from the remaining arguments. The term directly
represents the proof tree of the Elk proof. Intuitively, for the inner rule
applications, the result of the rule (its conclusion) is given as input to the surrounding
rule application. The term is a correct proof of P v 9r:U u R exactly if it is well
typed and has type h(P v 9 r U u R). Using the type ascription operator :, the
check command directs the LFSC checker to verify that.</p>
      <p>
        Extracting Proofs Using Elk's Explanation Servive
To generate LFSC proofs that can serve as certi cates, we used the explanation
service implemented in Elk 0.5 [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ]. Ideally, these certi cates would be created
during classi cation and returned together with the classi cation result to the
user. As a rst step towards this, we used the proof extraction methods that
are provided by Elk 0.5 [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] and can be direcly used from within Java. While
Kazakov et al. [
        <xref ref-type="bibr" rid="ref18">18</xref>
        ] describe algorithms for extracting proofs that could serve
as certi cates, the implementation available in Elk 0.5 is tailored towards the
explanation service within the OWL ontology editor Protege,7 which presents
some limitations for our use case. The aim of the explanation service in Protege
is to provide users with detailed explanations of the di erent inference steps
that could lead to a particular concept inclusion. If an axiom can be inferred in
di erent ways using Elk, the front-end shows all these possibilities. Users can
select the inference they are interested in and ask for inferences of each of the
premises, thus exploring di erent proofs step-wise for the concept inclusion of
interest. In a similar way, we use this service to reconstruct a proof automatically:
for a given axiom derived by Elk, we can ask for the possible rule applications
that have this axiom as conclusion. To construct the whole proof, we select one
such rule application, and then iteratively continue on the premises. A naive
implementation of this approach would lead to termination problems, since the
same axiom may occur twice in the resulting structure.
      </p>
      <p>This non-termination issue is solved in Algorithm 1 by keeping track, in
the variable inferences, of the set of inferences (i.e., rule applications including
7 https://protege.stanford.edu/
Algorithm 1 Algorithm used for generating proofs.
premises and conclusion) for the current branch of the proof being constructed.
To generate a proof of an axiom , we start with the call generateProof( ; ;) with
an empty set of inferences. Then we ask Elk via getInferencesFor( ) for inference
steps, but consider only those inferences that are not already in inferences. For
each inference inf, we call the procedure recursively on the premises involved,
adding the current inference to the set inferences (Line 8). If the proof
construction did not fail for any premise, we have constructed the proof and can return
it (Line 12). To further speed up the computation, we use a global cache to store
previously constructed proofs or sub-proofs (Line 1, 4, 5 and 11).
4</p>
    </sec>
    <sec id="sec-4">
      <title>Evaluation</title>
      <p>We used our approach to generate certi cates for E L+? ontologies and veri ed
them using LFSC. Our experiments, including classi cation, certi cate
generation, and certi cation, were performed on an Intel(R) Core(TM) i5-4590 CPU
with 4 cores at 3.30GHz and 32 GB RAM. The operating system was Debian
GNU/Linux 9. The code is available online.8
+ ontologies:</p>
      <p>
        We generated classi cation certi cates for two corpora of E L?
the OWL EL classi cation track of the OWL Reasoner Evaluation 2015 (ORE
2015) [
        <xref ref-type="bibr" rid="ref26">26</xref>
        ], and the Manchester OWL Corpus (MOWLCorp) [
        <xref ref-type="bibr" rid="ref21">21</xref>
        ]. From the
corpora, we removed all ontologies that (1) contained axioms not corresponding
+-axioms; or (2) were inconsistent. These two
to (syntactical variants of) E L?
cases only applied to ontologies in MOWLCorp. For each ontology, we used Elk
to generate the transitive reduct of the classi cation result, that is, we only
included strict subsumptions that could not be derived via transitivity from
8 https://lat.inf.tu-dresden.de/dl2020-certifying-classi cation-results
      </p>
      <p>ORE</p>
      <p>MOWLCorp
105
s 104
t
n
e
lm 103
i
a
t
n 102
E
#
101
100</p>
      <p>104
others, while also taking care that equivalence classes of concepts stayed intact.
Speci cally, for each such equivalence class, we would pick an arbitrary cyclic
chain of subsumption relations to be included. Furthermore, we excluded
concept inclusions/equivalences that were explicitly stated in the ontology. In the
following, when talking of the classi cation result, we always refer to the set
of inferred subsumption relationships obtained this way, that is, the transitive
reduct without explicitly stated concept inclusions and equivalences. We also
removed ontologies for which the transitive closure of the stated subsumptions
produced the whole hierarchy. After these removals, the ORE corpus contains 62
ontologies and the MOWLCorp corpus contains 310 ontologies. Figure 5 shows
the number of axioms and the number of entailments to be veri ed.</p>
      <p>We computed certi cates for each classi cation result and veri ed them with
the LFSC checker. For one of the ontologies from MOwlCorp, we terminated
the certi cate generation after 16 hours, while for all ontologies in ORE certi
cate was generated. In Figure 6, we show for each of the ontologies the time
taken for classi cation, certi cate generation, and veri cation by LFSC, where
the x-axis shows the number of entailments that was certi ed. One can see that
certi cate generation took signi cantly longer than the classi cation task itself,
while certi cate veri cation took signi cantly less time in almost all cases. The
long time for certi cate generation is to due the backtracking approach used
to reconstruct the proofs with Elk. We expect this to incur a much smaller
overhead if certi cates were generated directly during reasoning. It is therefore
more insightful to look at the sizes of the generated certi cates. In Figure 7,
we plot those sizes against the number of entailments in each ontology. Most
certi cates had a size between 1 KB and 1 MB, with the largest certi cate being
1.43 GB. Note that these are certi cates for the whole subsumption hierarchy,
and not for single subsumptions. Certi cates for single subsumptions are quite
small. Despite their large sizes, the certi cates were veri ed by the LFSC checker
in the order of milliseconds. Since we generated proofs for each entailment
separately, our classi cation certi cates collectively contained a lot of redundancy.
By caching and sharing subproofs, the cumulative size of the certi cates for all
the entailments could be signi cantly reduced.
5</p>
    </sec>
    <sec id="sec-5">
      <title>Conclusion</title>
      <p>We have implemented a prototypical proof certi cation system for the reasoner
Elk, utilizing the proof-generation facilities available in Elk for the generation
of certi cates, and employing the LFSC checker as proof-checker. Our
evaluation demonstrates that, even though certi cates may be quite large, they can
nevertheless be veri ed in very short time. We exploited the explanation
service of Elk to generate proof certi cates a posteriori. Ideally, the generation
of certi cates should take place during reasoning. This would reduce certi cate
generation times considerably and result in smaller certi cates. Before
embarking on the major task of implementing a proof-producing reasoner for E L, we
found it sensible to assess rst whether proof checking of DL reasoning results
based on LFSC is viable in principle.</p>
      <p>
        Since our focus was on evaluating certi cate checking rather than certi cate
generation, our algorithm for extracting proofs from Elk is fairly unsophisticated
and not optimal. For instance, by building the proof starting from the relevant
axioms in the ontology, rather than from the conclusion, we would avoid the need
for detecting cycles. This should lead to a procedure with the same (polynomial)
time complexity as Elk. The Proof Utility Library PULi9 or the techniques by
Alrabbaa et al. [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ] for extracting proofs of minimal size from Elk could be
of further help in implementing a more e cient certi cate generation service.
Nevertheless, note that the size of certi cates and the time required to generate
them constitutes a bottleneck only if one intends to certify all reasoning results
(i.e., the whole subsumption hierarchy). The current unoptimized approach to
certi cation could already be useful in cases where a speci c subsumption result
by a DL reasoner is called into question; for example, if the user doubts it, or if
it di ers from the result produced by another reasoner.
      </p>
      <p>
        Certi cation of reasoning results is even more important for more
expressive DLs than the one supported by Elk since reasoners for them are more
complex and so more likely to contain errors. There are reasoners for more
expressive DLs, such as Avalance [
        <xref ref-type="bibr" rid="ref35">35</xref>
        ], Konklude [
        <xref ref-type="bibr" rid="ref30">30</xref>
        ] and Sequoia [
        <xref ref-type="bibr" rid="ref12">12</xref>
        ], that use
consequence-based reasoning similarly to Elk, sometimes in combination with
other techniques. For such reasoners, our approach should be relatively easy to
adapt. To the best of our knowledge, none of the tableau-based DL reasoners
generate proofs of their computed consequences, though there was some early
work on how to extract sequent proofs from a run of a tableaux algorithm [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ].
      </p>
      <p>Encoding complex proof systems in LFSC has the cost of trusting, or proving
formally, the soundness of the encoded proof rules for each system. An
alternative approach would be to design a standardised, simpler proof system for all
DLs, and have reasoners translate their proofs into proofs in that system. The
challenge in that case is probably more of a social than a technical nature as it
requires a community-wide acceptance of a common standard.</p>
      <p>Finally, we point out that proofs, and proof checkers, can be used only to
validate the soundness of a classi cation result. In principle, the completeness of
classi cation results can be certi ed if the reasoner provides counter-examples
for non-entailed subsumption relations, which could then be validated using
model checking techniques for rst-order logic. Because the number of
nonsubsumptions is usually much higher than the number of subsumptions, the
challenge in this case would be nding common certi cates for a large number
of non-subsumptions at a time.
9 https://github.com/liveontologies/puli</p>
    </sec>
    <sec id="sec-6">
      <title>LFSC signature for Elk's calculus</title>
      <p>Included below is the LFSC signature used to model the fragment of the calculus
of Elk used for our experiments. The source code of our implementation, CSV
les with evaluation results, and information on how to repeat the experiments
are provided under</p>
      <p>https://lat.inf.tu-dresden.de/dl2020-certifying-classi cation-results .</p>
      <p>For space constraints, the paper glosses over a number of lower level technical
points. For instance, the handling of role chains is more complex in practice, and
an exact description would have been out of the scope of the paper. The main
points are the following.</p>
      <p>{ LFSC does not support rules with a variable number of premises, which is
why we had to represent rule applications of this kind by a sequence of rule
application with a xed number of premises. For the -rule, the easiest way
to do this is to allow existential role restrictions to contain role chains instead
of roles, as we can then build up the role chain step-wise, before matching
it with the role inclusion axiom.
{ For certi cation purposes, it is more convenient to use the needed axioms
from the ontology as proof assumptions, instead of having side conditions
that check their membership in the ontology. Checking that proof
assumptions are from a given ontology can be achieved with the TBox proof rule.
{ Types rname, rchain, concept, axiom, and ontology are used respectively for
roles, role chains, concepts, axioms, and ontologies.
{ An LFSC signature is de ned by a series of commands for declaring or
de ning constants, for checking the well-typedness of term, and for de
ning side-condition programs. New type or term constants are introduced by
the declare command. Side condition programs are de ned by the program
command.
{ The concrete syntax of LFSC is s-expression-based, as in Lisp and Scheme,
so all the encoded E L operators are pre x operators (and typically have
an alphanumeric name). Comments in the signature (lines starting with ;)
relate these operators to the E L ones they encode.
{ While we do not use TBox axioms as side conditions in the certi cation, side
conditions are used for various technical reasons not discussed in the paper.
For instance, they are used to compute the concatenation of two role chains.
The actual LFSC signature used in the experiments follows next.
;; LFSC encoding of ELK proof system
;; Authors: Patrick Koopmann and Cesare Tinelli
;; Date: June 2020
;-------; Legend
;-------; (t1 t2)
; (! x 1 2)
; (\ x t)
; (% x t)
; (@ x t e)
; (: t)
denotes function application
" x : 1: 2
" x:t
" x : :t
" let x = t in e
" t :
;; Note: Except for function applications, the syntax above is
;; used in LFSC terms, not in side condition programs
;---------------; Proof Language
;--------------;; Note: Every user-defined type
;; an algebraic datatype:
;; every function of return type
;;
; Role name type
(declare rname type)
;; Note: no constructors are needed for rname,
;; using variables of type rname is enough.</p>
      <p>in LFSC is effectively
is a constructor for .
; Role chain type
(declare rchain type)
;; Note: rchain is used to represent concatenations of roles
;; in (flat) normal form.
; Role chain constructors
;
; empty role chain
(declare erc rchain)
; non-empty role chain constructor
(declare rc (! x rname (! y rchain rchain)))
;; Note: rchains are essentially erc-terminated lists.
; Concept type
(declare concept type)
; Concept constructors
;
; atomic concept (has the form (ac 0), (ac 1), ...)
(declare ac (! n mpz concept))
;; Note: mpz is the builtin type in LFSC of (infinite precision)
;; integers. The usual integer operators are built in.
;
; &gt;
(declare top concept)
; ?
(declare bot concept)
; _ u _
(declare inter (! x concept (! y concept concept)))
; 9 _._
(declare ex (! x rchain (! y concept concept)))
;; Note: The first argument of ex is a *role chain*, not a role.
;; This is for convenience.
;
;; : _
(declare neg (! c concept concept))
;; Note : should not be part of EL, but interestingly,
;; some rules of ELK use it.
; Axiom type
(declare axiom type)
; Axiom constructors
;
; Concept inclusion _ v _
(declare sub (! x concept (! y concept axiom)))
; Concept equivalence _ _
(declare eq (! x concept (! y concept axiom)))
; Role (chain) inclusion _ v _
(declare rsub (! x rchain (! y rchain axiom)))
; Role (chain) equivalence _ _
(declare req (! x rchain (! y rchain axiom)))
; object property domain axioms
(declare roleDomain (! x rchain (! y concept axiom)))
; object property range axioms
(declare roleRange (! x rchain (! y concept axiom)))
; transitive property axiom
(declare transRole (! r rchain axiom))
; Ontology type
(declare ontology type)
; Ontology constructors
;
; empty ontology
(declare eon ontology)
; non-empty ontology constructor
(declare on (! x axiom (! y ontology ontology)))
;; Note: An ontology is essentially a (flat, eon-termindated)
;; list of of axioms.
;------------------------; Side Condition Programs
;------------------------;; Side condition terms can appear directly in a side condition
;; or can be abstracted in named, parametrized programs.
;; Programs are functional, monomorphic, and first-order.
;; They can be recursive but not mutually recursive. They can
;; diverge, terminate normally (returning a value), or
;; abnormally (raising a "fail" exception).
;; Inputs and output are LF terms which need not be ground,
;; that is, they can contain logical variables. Those are not
;; to be confused with program variables, i.e., input parameters
;; and local variables.
;; Local variables are introduced by the match and let
;; constructs. Program variables are lexically scoped.
;; Program evaluation is eager, with pass-by-value semantics.
; Useful for side condition programs
(declare bool type)
; bool constructors
(declare tt bool)
(declare ff bool)
; (IsIn t1 t2) returns tt if t1 occurs as an axiom of t2;
; it returns ff otherwise.
(program isIn ((a axiom) (o ontology)) bool
(match o
(eon ff)
((on a1 o1)
; evaluates to tt if a1 is syntactically equal to a;
; and to the value of (isIn a o1) otherwise
(ifequal a1 a tt (isIn a o1)))))
;; Note: For (IsIn t1 t2) to terminate normally t2 cannot be
;; a logical variable; it has to have the form eon or
;; (on a1 (on a2 ... (on an eon) ...)), otherwise match will fail.
;; However, t1, a1, ..., an can all be or contain logical variables.
; Concatenates two role chains (like list append)
(program concat ((rc1 rchain) (rc2 rchain)) rchain</p>
      <p>(match rc1
(erc rc2)
((rc r rs) (rc r (concat rs rc2)))))
;; Note: For similar reason as in isIn, (concat t1 t2) fails
;; if t1 is a logical variable
; Proof judgment for assumed or proved axioms.
; Technically, a type dependent on input axiom c.
(declare holds (! c axiom type))
;; The proof rules are functions whose return type has the form
;; (holds a) where a is the rule's conclusion.
;; The input parameters correspond to the rule's parameters
;; and premises, if any</p>
      <p>(! sc (^ (isIn a o) tt)
;
----------------------</p>
      <p>(holds a)
; TBox
; Every axiom in o is derivable
(declare TBox (! o ontology (! a axiom ; rule parameters
; ; premises (none)
; side condition
; conclusion
;; Note: For notational convenience, side conditions are
;; introduced as fake arguments whose "type" has the form
;; (^ p r) where
;; - p is a side condition program (a term in the side-condition
;; language) and
;; - r is a term (possibly with variables).
;; When a proof rule is applied to actual arguments,
;; the side condition succeeds iff
;; 1) running p doesn't cause a fail exception and
;; 2) the result of p matches r
;; Any unbound variables in r get bound by the matching
;; substitution.
;; In TBox, p is the call (isIn a o) and r is the ground Boolean
;; term tt.
;; Note: The term (TBox t1 t2) is well typed iff
;; - t1 has type ontology,
;; - t2 has type axiom, and
;; - (isIn t1 t2) evaluates to tt
;; Since the the formal parameter sc of TBox is fictitious,
;; no actual parameter for it is needed in the rule application
;; (TBox t1 t2).
; Bottom Subclass
(declare Bottom (! c concept ; rule parameters
; ; premises (none)
;
------------------(holds (sub bot c)) ; conclusion
? v C
;
))
))
; Top Superclass
(declare Top (! c concept
;
;--------------------</p>
      <p>(holds (sub c top))
; Class Inclusion Tautology
(declare subRef (! c concept
;
;
----------------(holds (sub c c))
; C v C</p>
      <p>; C v &gt;
;
))
;
))
; Property Inclusion Tautology
(declare rsubRef (! r rchain
;
;
-----------------(holds (rsub r r))</p>
      <p>; r v r
; Intersection Decomposition 1
(declare InterDec1 (! c concept (! d concept
;
;
--------------------------</p>
      <p>(holds (sub (inter c d) c)) ; C u D v C
; Intersection Decomposition
; Useful for nested intersections
(declare InterDec (! c concept (! d concept
;</p>
      <p>(! sc (^ (icontains c d) tt) ; D occurs in C
;
---------------------------</p>
      <p>(holds (sub c d)) ; C v D
; Intersection Composition
(declare InterComp (! c concept (! d concept (! e concept
;
;
)))
;
)))
;
))</p>
      <p>(! p (holds (sub c d)) ; C v D
;
------------------------------(holds (sub (ex r c) (ex r d))) ; 9r:C v 9r:D
(! p1 (holds (sub c d))
(! p2 (holds (sub c e))
;
--------------------------</p>
      <p>(holds (sub c (inter d e)))
;
))))))
; Existential of Bottom
(declare ExBottom (! r rchain
;
;
---------------------------(holds (sub (ex r bot) bot)) ; 9r:? v ?
; C v D
; C v E
; C v D u E
; Existential Property Expansion
(declare ExExpandRole (! c concept (! r1 rchain (! r2 rchain
;</p>
      <p>(! p (holds (rsub r1 r2))
;
--------------------------------</p>
      <p>(holds (sub (ex r1 c) (ex r2 c)))
)))))
; r1 v r2
; 9r1:C v 9r2:C
;; Note: The Existential Composition rule of ELK cannot be
;; encoded as a single LFSC rule because it has a variable
;; number of premises. So it has by several variants in LFSC.
; Existential Composition 1
(declare ExComp1 (! c0 concept (! c1 concept (! c2 concept</p>
      <p>(! r1 rchain (! r2 rchain (! r rchain
;
;
;
;
)))))))))))
; Existential Composition 2
(declare ExComp2 (! c0 concept (! c1 concept</p>
      <p>(! r1 rchain (! r rchain
(! p1 (holds (sub c0 (ex r1 c1)))
(! p2 (holds (rsub r1 r))
;
--------------------------------</p>
      <p>(holds (sub c0 (ex r c1)))
;
)))))))
; C0 v 9r1:C1
; C1 v 9r2:C2
; r = r1 r2
; C0 v 9r:C2
; C0 v 9r1:C1
; r1 v r
; C0 v 9r:C1
; Existential Composition
(declare ExComp (! c0 concept (! c1 concept (! c2 concept</p>
      <p>(! r1 rchain (! r2 rchain (! r12 rchain (! r rchain
; Class Hierarchy
(declare ConHi (! c1 concept (! c2 concept (! c3 concept
;
;
))))))
; C1 v C2
; C2 v C3
; C1 v C3
; Property Hierarchy
(declare RolHi (! r1 rchain (! r2 rchain (! r3 rchain
;
; Equivalent Classes Decomposition 1
(declare EqDec1 (! c concept (! d concept
;</p>
      <p>(! p (holds (eq c d))
;
--------------------</p>
      <p>(holds (sub c d))
;
))))
; Equivalent Classes Decomposition 2
(declare EqDec2 (! c concept (! d concept
;</p>
      <p>(! p (holds (eq c d))
;
--------------------</p>
      <p>(holds (sub d c))
;
))))
; C</p>
      <p>D
; C v D
; C</p>
      <p>D
; D v C
; Classes Inclusion Cycle
(declare ConCyc (! c concept (! d concept
;
(! p1 (holds (sub c d))
(! p2 (holds (sub d c))
;
----------------------</p>
      <p>(holds (eq c d))
;
)))))
; C v D
; D v C
; D</p>
      <p>C
; Proof judgment used to single out a goal.
(declare goal (! c axiom type))
; Proof judgment used for convenience.
; All goal-oriented proofs end in done.
(declare done type)
(! p1 (goal g)
(! p2 (holds g)
;
--------------</p>
      <p>done
))))
; we can conclude that we are done
;---------------------------------------------------------; Missing Rules used by ELK but not mentioned in ELK paper
;---------------------------------------------------------; Property Domain Translation
(declare RoleDomain (! r rchain (! c concept
;</p>
      <p>(! p1 (holds (roleDomain r c))
;
-----------------------------</p>
      <p>(holds (sub (ex r top) c))
))))
; The domain of r is C
; 9r:&gt; v C
; Transitive Role
(declare TransitiveRole (! r rchain (! rr rchain
;
(! p1 (holds (transRole r))
(! sc (^ (concat r r) rr)
;
--------------------------</p>
      <p>(holds (rsub rr r))
)))))
; Negation Clash
(declare NegationClash (! c concept
;
;
---------------------------------(holds (sub (inter c (neg c)) bot))
))
; r is transitive
; rr = r r
; rr v r
; C u :C v ?</p>
    </sec>
    <sec id="sec-7">
      <title>Sample LFSC certi cates</title>
      <p>Included below is a le with several proof certi cates The certi cates were
manually generated for greater readability. A few of the certi cates automatically
generated with our implementation are available at
https://lat.inf.tu-dresden.de/dl2020-certifying-classi cation-results.
;--------------------------------------------; Sample proof certificates for ELK signature
;--------------------------------------------; check proof that &gt; v &gt;
(check
(: (holds (sub top top))</p>
      <p>(InterDec top top)
))
; check proof that C u D v C
(check
(% C concept
(% D concept
(: (holds (sub (inter C D) C))</p>
      <p>(InterDec (inter C D) C)
)))))
; check proof that C u D v D
(check
(% n1 mpz
(% n2 mpz
(@ C (ac n1)
(@ D (ac n2)
(: (holds (sub (inter C D) D))</p>
      <p>(InterDec (inter C D) D)
))))))
; Deductive proof of Kidney v Kidney
; Shows an axiom derivable from no assumption
(check
(% Kidney concept
(: (holds (sub Kidney Kidney))</p>
      <p>(subRef Kidney)
)))
; Goal oriented proof of Kidney v Kidney
; the expected goal is specified beforehand
(check
;
-----------------------------------------------------;; Note:: _ above can be used instead of the actual
;; parameter whenever the latter can be constructed
;; from the other actual parameters by type inference.
;; In this case, the inferred actual parameter is
;; (sub Kidney Kidney)
; Proof from ontology
;
; Check that AntiDiuresis v 9isFunctionOf.Kidney
; follows from an ontology containing
; - AntiDiuresis (Excretion u 9actsSpecificallyOn.Urine
; u 9hasProcessActivity.decreasedActivityLevel)
; - AntiDiuresis v ExcretionOfUrine
; - ExcretionOfUrine v 9isFunctionOf.Kidney
(check
; Roles
(% actsSpecificallyOn rname
(% hasProcessActivity rname
(% isFunctionOf rname
; Concepts
(@ AntiDiuresis (ac 1)
(@ decreasedActivityLevel (ac 2)
(@ Excretion (ac 3)
(@ ExcretionOfUrine (ac 4)
(@ Kidney (ac 5)
(@ Urine (ac 6)
; Axioms
; AntiDiuresis (Excretion u 9actsSpecificallyOn.Urine
; u 9hasProcessActivity.decreasedActivityLevel)
(@ a1 (eq AntiDiuresis
(inter Excretion
(inter (ex (rc actsSpecificallyOn erc) Urine)</p>
      <p>(ex (rc hasProcessActivity erc) decreasedActivityLevel))))
; AntiDiuresis v ExcretionOfUrine
(@ a2 (sub AntiDiuresis ExcretionOfUrine)
; ExcretionOfUrine v 9isFunctionOf.Kidney
(@ a3 (sub ExcretionOfUrine (ex (rc isFunctionOf erc) Kidney))
; Ontology
(@ o (on a1 (on a2 (on a3 eon)))
; Goal: AntiDiuresis v 9isFunctionOf.Kidney
(% g (goal (sub AntiDiuresis (ex (rc isFunctionOf erc) Kidney)))
; Assumptions
; Proof of goal from assumptions
(: done (Proved _ g
(ConHi _ _ _
(TBox o a2)
(TBox o a3)
)
))))))))))))))))))
;; Note: the applications of TBox are not really needed.
;; A proof of g from explicit assumptions a2 and a3
;; should suffice as a proof certificate.
;; Proof from assumptions (no TBox applications)
(check
; Roles
(% actsSpecificallyOn rname
(% hasProcessActivity rname
(% isFunctionOf rname
; Concepts
(@ AntiDiuresis (ac 1)
(@ decreasedActivityLevel (ac 2)
(@ Excretion (ac 3)
(@ ExcretionOfUrine (ac 4)
(@ Kidney (ac 5)
(@ Urine (ac 6)
; Goal: AntiDiuresis v 9isFunctionOf.Kidney
(% g (goal (sub AntiDiuresis (ex (rc isFunctionOf erc) Kidney)))
; Assumptions
; AntiDiuresis (Excretion u 9actsSpecificallyOn.Urine
; u 9hasProcessActivity.decreasedActivityLevel)
(% p1 (holds (eq AntiDiuresis
(inter Excretion
(inter (ex (rc actsSpecificallyOn erc) Urine)
(ex (rc hasProcessActivity erc)</p>
      <p>decreasedActivityLevel)))))
; (Excretion u 9actsSpecificallyOn.Urine) ExcretionOfUrine
(% p2 (holds (eq (inter Excretion</p>
      <p>(ex (rc actsSpecificallyOn erc) Urine))</p>
      <p>ExcretionOfUrine))
; ExcretionOfUrine v 9isFunctionOf.Kidney
(% p3 (holds (sub ExcretionOfUrine</p>
      <p>(ex (rc isFunctionOf erc) Kidney)))
; Proof of goal from assumptions
(: done
(Proved _ g
(ConHi _ _ _
(ConHi _ _ _
(InterComp _ _ _
(ConHi _ _ _
(EqDec1 _ _ p1)
(InterDec1
Excretion ; actually inferrable, kept for readability
(inter (ex (rc actsSpecificallyOn erc) Urine)
(ex (rc hasProcessActivity erc)</p>
      <p>decreasedActivityLevel))))
(ConHi _ _ _
(EqDec1 _ _ p1)
(InterDec
(inter Excretion
(inter (ex (rc actsSpecificallyOn erc) Urine)
(ex (rc hasProcessActivity erc)</p>
      <p>decreasedActivityLevel)))
(ex (rc actsSpecificallyOn erc) Urine))))
(EqDec1 _ _ p2))
p3)))
))))))))))))))</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          1.
          <string-name>
            <surname>Alrabbaa</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Baader</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Borgwardt</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Koopmann</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kovtunova</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>Finding small proofs for description logic entailments: Theory and practice</article-title>
          . In: Albert,
          <string-name>
            <given-names>E.</given-names>
            ,
            <surname>Kovacs</surname>
          </string-name>
          ,
          <string-name>
            <surname>L</surname>
          </string-name>
          . (eds.)
          <source>LPAR</source>
          <year>2020</year>
          :
          <article-title>23rd International Conference on Logic for Programming, Arti cial Intelligence and Reasoning</article-title>
          . EPiC Series in Computing, vol.
          <volume>73</volume>
          , pp.
          <volume>32</volume>
          {
          <fpage>67</fpage>
          .
          <string-name>
            <surname>EasyChair</surname>
          </string-name>
          (
          <year>2020</year>
          ), https://easychair.org/publications/paper/qgX6
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          2.
          <string-name>
            <surname>Armand</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Faure</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gregoire</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          , Keller, C.,
          <string-name>
            <surname>Thery</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Werner</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          :
          <article-title>A modular integration of SAT/SMT solvers to Coq through proof witnesses</article-title>
          . In: Jouannaud,
          <string-name>
            <given-names>J.</given-names>
            ,
            <surname>Shao</surname>
          </string-name>
          ,
          <string-name>
            <surname>Z</surname>
          </string-name>
          . (eds.)
          <source>Proceedings of the First International Conference on Certi ed Programs and Proofs, CPP 2011. Lecture Notes in Computer Science</source>
          , vol.
          <volume>7086</volume>
          , pp.
          <volume>135</volume>
          {
          <fpage>150</fpage>
          . Springer (
          <year>2011</year>
          ). https://doi.org/10.1007/978-3-
          <fpage>642</fpage>
          -25379-9 12
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          3.
          <string-name>
            <surname>Assaf</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Burel</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Cauderlier</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Delahaye</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dowek</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dubois</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gilbert</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Halmagrand</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hermant</surname>
            ,
            <given-names>O.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Saillard</surname>
          </string-name>
          , R.:
          <article-title>Expressing theories in the - calculus modulo theory and in the Dedukti system</article-title>
          . In: Ghilezan,
          <string-name>
            <given-names>S.</given-names>
            ,
            <surname>Geuvers</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            ,
            <surname>Ivetic</surname>
          </string-name>
          ,
          <string-name>
            <surname>J</surname>
          </string-name>
          . (eds.)
          <source>Proceedings of the 22nd International Conference on Types for Proofs and Programs</source>
          ,
          <string-name>
            <surname>TYPES</surname>
          </string-name>
          <year>2016</year>
          . vol.
          <volume>97</volume>
          .
          <string-name>
            <surname>Novi</surname>
            <given-names>SAd</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Serbia</surname>
          </string-name>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          4.
          <string-name>
            <surname>Baader</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Brandt</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lutz</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Pushing the EL envelope</article-title>
          . In: Kaelbling,
          <string-name>
            <given-names>L.P.</given-names>
            ,
            <surname>Sa</surname>
          </string-name>
          <string-name>
            <surname>otti</surname>
          </string-name>
          ,
          <source>A. (eds.) Proc. of the 19th Int. Joint Conf. on Arti cial Intelligence (IJCAI</source>
          <year>2005</year>
          ). pp.
          <volume>364</volume>
          {
          <fpage>369</fpage>
          . Morgan Kaufmann, Los Altos, Edinburgh (UK) (
          <year>2005</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          5.
          <string-name>
            <surname>Baader</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Franconi</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hollunder</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nebel</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Pro</surname>
            <given-names>tlich</given-names>
          </string-name>
          ,
          <string-name>
            <surname>H.J.:</surname>
          </string-name>
          <article-title>An empirical analysis of optimization techniques for terminological representation systems or: Making KRIS get a move on</article-title>
          .
          <source>Applied Arti cial Intelligence</source>
          .
          <source>Special Issue on Knowledge Base Management</source>
          <volume>4</volume>
          ,
          <issue>109</issue>
          {
          <fpage>132</fpage>
          (
          <year>1994</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          6.
          <string-name>
            <surname>Baader</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Horrocks</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Lutz</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sattler</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          :
          <article-title>An Introduction to Description Logic</article-title>
          . Cambridge University Press (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          7.
          <string-name>
            <surname>Baader</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kriegel</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Nuradiansyah</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          , Pen~aloza, R.:
          <article-title>Making repairs in description logics more gentle</article-title>
          . In: Thielscher,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Toni</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            ,
            <surname>Wolter</surname>
          </string-name>
          ,
          <string-name>
            <surname>F</surname>
          </string-name>
          . (eds.)
          <source>Proc. of the Sixteenth International Conference on Principles of Knowledge Representation and Reasoning (KR</source>
          <year>2018</year>
          ). pp.
          <volume>319</volume>
          {
          <fpage>328</fpage>
          . AAAI Press (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          8.
          <string-name>
            <surname>Barrett</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>de Moura</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fontaine</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Proofs in satis ability modulo theories</article-title>
          . In: Delahaye,
          <string-name>
            <surname>D.</surname>
          </string-name>
          ,
          <string-name>
            <given-names>Woltzenlogel</given-names>
            <surname>Paleo</surname>
          </string-name>
          , B. (eds.)
          <article-title>All about Proofs, Proofs for All</article-title>
          ,
          <source>Mathematical Logic and Foundations</source>
          , vol.
          <volume>55</volume>
          , pp.
          <volume>23</volume>
          {
          <fpage>44</fpage>
          .
          <string-name>
            <surname>College</surname>
            <given-names>Publications</given-names>
          </string-name>
          , London, UK (Jan
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          9.
          <string-name>
            <surname>Blanchette</surname>
            ,
            <given-names>J.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kaliszyk</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Paulson</surname>
            ,
            <given-names>L.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Urban</surname>
          </string-name>
          , J.:
          <article-title>Hammering towards QED</article-title>
          .
          <source>J. Formalized Reasoning</source>
          <volume>9</volume>
          (
          <issue>1</issue>
          ),
          <volume>101</volume>
          {
          <fpage>148</fpage>
          (
          <year>2016</year>
          ). https://doi.org/10.6092/issn.1972-
          <volume>5787</volume>
          /
          <fpage>4593</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          10.
          <string-name>
            <surname>Borgida</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Franconi</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Horrocks</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>McGuinness</surname>
            ,
            <given-names>D.L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Patel-Schneider</surname>
            ,
            <given-names>P.F.</given-names>
          </string-name>
          :
          <article-title>Explaining ALC subsumption</article-title>
          . In: Horn,
          <string-name>
            <surname>W</surname>
          </string-name>
          . (ed.)
          <source>Proc. of the 14th Eur. Conf. on Arti cial Intelligence (ECAI</source>
          <year>2000</year>
          ). pp.
          <volume>209</volume>
          {
          <fpage>213</fpage>
          . IOS Press (
          <year>2000</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          11.
          <string-name>
            <surname>Bouton</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oliveira</surname>
          </string-name>
          , D.
          <string-name>
            <surname>C.B.D.</surname>
          </string-name>
          ,
          <string-name>
            <surname>Deharbe</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Fontaine</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          : veriT:
          <article-title>An open, trustable and e cient SMT-solver</article-title>
          . In: Schmidt,
          <string-name>
            <surname>R.A</surname>
          </string-name>
          . (ed.)
          <source>Proceedings of the 22nd International Conference on Automated Deduction, CADE-22. Lecture Notes in Computer Science</source>
          , vol.
          <volume>5663</volume>
          , pp.
          <volume>151</volume>
          {
          <fpage>156</fpage>
          . Springer (
          <year>2009</year>
          ). https://doi.org/10.1007/978-3-
          <fpage>642</fpage>
          -02959-2 12
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          12.
          <string-name>
            <surname>Cucala</surname>
            ,
            <given-names>D.T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Grau</surname>
            ,
            <given-names>B.C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Horrocks</surname>
            ,
            <given-names>I.</given-names>
          </string-name>
          :
          <article-title>Sequoia: A consequence based reasoner for SROIQ</article-title>
          . In: Simkus,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Weddell</surname>
          </string-name>
          , G.E. (eds.)
          <source>Proceedings of the 32nd International Workshop on Description Logics</source>
          , Oslo, Norway, June 18-21,
          <year>2019</year>
          .
          <source>CEUR Workshop Proceedings</source>
          , vol.
          <volume>2373</volume>
          .
          <string-name>
            <surname>CEUR-WS.org</surname>
          </string-name>
          (
          <year>2019</year>
          ), http://ceur-ws.
          <source>org/</source>
          Vol-
          <volume>2373</volume>
          /paper-27.pdf
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          13.
          <string-name>
            <surname>Hadarean</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Barrett</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Reynolds</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tinelli</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Deters</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Fine grained SMT proofs for the theory of xed-width bit-vectors</article-title>
          . In: Davis,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Fehnker</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            ,
            <surname>McIver</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            ,
            <surname>Voronkov</surname>
          </string-name>
          ,
          <string-name>
            <surname>A</surname>
          </string-name>
          . (eds.)
          <source>Proceedings of the 20th International Conference on Logic for Programming</source>
          ,
          <source>Arti cial Intelligence, and Reasoning</source>
          (Suva, Fiji).
          <source>Lecture Notes in Computer Science</source>
          , vol.
          <volume>9450</volume>
          , pp.
          <volume>340</volume>
          {
          <fpage>355</fpage>
          . Springer (
          <year>2015</year>
          ). https://doi.org/10.1007/978-3-
          <fpage>662</fpage>
          -48899-7 24
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          14.
          <string-name>
            <surname>Harper</surname>
            ,
            <given-names>R.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Honsell</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Plotkin</surname>
          </string-name>
          , G.:
          <article-title>A Framework for De ning Logics</article-title>
          .
          <source>Journal of the Association for Computing Machinery</source>
          <volume>40</volume>
          (
          <issue>1</issue>
          ),
          <volume>143</volume>
          {184 (Jan
          <year>1993</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          15.
          <string-name>
            <surname>Horridge</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sattler</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          :
          <article-title>Laconic and precise justi cations in OWL</article-title>
          . In: Sheth,
          <string-name>
            <given-names>A.P.</given-names>
            ,
            <surname>Staab</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            ,
            <surname>Dean</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Paolucci</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Maynard</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            ,
            <surname>Finin</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.W.</given-names>
            ,
            <surname>Thirunarayan</surname>
          </string-name>
          ,
          <string-name>
            <surname>K</surname>
          </string-name>
          . (eds.) 7th
          <source>International Semantic Web Conference(ISWC 2008). Lecture Notes in Computer Science</source>
          , vol.
          <volume>5318</volume>
          , pp.
          <volume>323</volume>
          {
          <fpage>338</fpage>
          . Springer-Verlag (
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          16.
          <string-name>
            <surname>Kalyanpur</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sirin</surname>
            ,
            <given-names>E.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Grau</surname>
            ,
            <given-names>B.C.</given-names>
          </string-name>
          :
          <article-title>Repairing unsatis able concepts in OWL ontologies</article-title>
          . In: Sure,
          <string-name>
            <given-names>Y.</given-names>
            ,
            <surname>Domingue</surname>
          </string-name>
          ,
          <string-name>
            <surname>J</surname>
          </string-name>
          . (eds.)
          <source>Proc. of the 3rd Eur. Semantic Web Conference (ESWC'06). Lecture Notes in Computer Science</source>
          , vol.
          <volume>4011</volume>
          , pp.
          <volume>170</volume>
          {
          <fpage>184</fpage>
          . Springer-Verlag (
          <year>2006</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref17">
        <mixed-citation>
          17.
          <string-name>
            <surname>Kazakov</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          :
          <article-title>Consequence-driven reasoning for Horn SHIQ ontologies</article-title>
          . In: Boutilier,
          <string-name>
            <surname>C</surname>
          </string-name>
          . (ed.)
          <source>Proc. of the 21st Int. Joint Conf. on Arti cial Intelligence (IJCAI</source>
          <year>2009</year>
          ). pp.
          <year>2040</year>
          {
          <year>2045</year>
          .
          <article-title>IJCAI/AAAI (</article-title>
          <year>2009</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref18">
        <mixed-citation>
          18.
          <string-name>
            <surname>Kazakov</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Klinov</surname>
            ,
            <given-names>P.</given-names>
          </string-name>
          :
          <article-title>Goal-directed tracing of inferences in EL ontologies</article-title>
          . In: Mika,
          <string-name>
            <given-names>P.</given-names>
            ,
            <surname>Tudorache</surname>
          </string-name>
          ,
          <string-name>
            <given-names>T.</given-names>
            ,
            <surname>Bernstein</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            ,
            <surname>Welty</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            ,
            <surname>Knoblock</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.A.</given-names>
            ,
            <surname>Vrandecic</surname>
          </string-name>
          ,
          <string-name>
            <given-names>D.</given-names>
            ,
            <surname>Groth</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.T.</given-names>
            ,
            <surname>Noy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.F.</given-names>
            ,
            <surname>Janowicz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>K.</given-names>
            ,
            <surname>Goble</surname>
          </string-name>
          ,
          <string-name>
            <surname>C.A</surname>
          </string-name>
          . (eds.)
          <source>Proc. of the 13th International Semantic Web Conference (ISWC 2014). Lecture Notes in Computer Science</source>
          , vol.
          <volume>8797</volume>
          , pp.
          <volume>196</volume>
          {
          <fpage>211</fpage>
          . Springer (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref19">
        <mixed-citation>
          19.
          <string-name>
            <surname>Kazakov</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          , Krotzsch,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Simancik</surname>
          </string-name>
          ,
          <string-name>
            <surname>F.</surname>
          </string-name>
          :
          <article-title>The incredible ELK - from polynomial procedures to e cient reasoning with EL ontologies</article-title>
          .
          <source>J. Autom. Reasoning</source>
          <volume>53</volume>
          (
          <issue>1</issue>
          ),
          <volume>1</volume>
          {
          <fpage>61</fpage>
          (
          <year>2014</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref20">
        <mixed-citation>
          20.
          <string-name>
            <surname>Lee</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Matentzoglu</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Sattler</surname>
            ,
            <given-names>U.</given-names>
          </string-name>
          :
          <article-title>A multi-reasoner, justi cationbased approach to reasoner correctness</article-title>
          . In: Arenas,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Corcho</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            ,
            <surname>Simperl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>E.</given-names>
            ,
            <surname>Strohmaier</surname>
          </string-name>
          , M.,
          <string-name>
            <surname>d'Aquin</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Srinivas</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Groth</surname>
            ,
            <given-names>P.T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Dumontier</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          , He in, J.,
          <string-name>
            <surname>Thirunarayan</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Staab</surname>
          </string-name>
          , S. (eds.)
          <source>Proc. of the 14th International Semantic Web Conference (ISWC 2015). Lecture Notes in Computer Science</source>
          , vol.
          <volume>9367</volume>
          , pp.
          <volume>393</volume>
          {
          <fpage>408</fpage>
          . Springer (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref21">
        <mixed-citation>
          21.
          <string-name>
            <surname>Matentzoglu</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bail</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          :
          <article-title>A snapshot of the OWL Web</article-title>
          . In: Alani,
          <string-name>
            <given-names>H.</given-names>
            ,
            <surname>Kagal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            ,
            <surname>Fokoue</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            ,
            <surname>Groth</surname>
          </string-name>
          ,
          <string-name>
            <given-names>P.T.</given-names>
            ,
            <surname>Biemann</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            ,
            <surname>Parreira</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.X.</given-names>
            ,
            <surname>Aroyo</surname>
          </string-name>
          ,
          <string-name>
            <given-names>L.</given-names>
            ,
            <surname>Noy</surname>
          </string-name>
          ,
          <string-name>
            <given-names>N.F.</given-names>
            ,
            <surname>Welty</surname>
          </string-name>
          ,
          <string-name>
            <given-names>C.</given-names>
            ,
            <surname>Janowicz</surname>
          </string-name>
          ,
          <string-name>
            <surname>K</surname>
          </string-name>
          . (eds.)
          <source>The Semantic Web - ISWC 2013 - 12th International Semantic Web Conference</source>
          , Sydney,
          <string-name>
            <surname>NSW</surname>
          </string-name>
          , Australia,
          <source>October 21-25</source>
          ,
          <year>2013</year>
          , Proceedings,
          <source>Part I. Lecture Notes in Computer Science</source>
          , vol.
          <volume>8218</volume>
          , pp.
          <volume>331</volume>
          {
          <fpage>346</fpage>
          . Springer (
          <year>2013</year>
          ). https://doi.org/10.1007/978-3-
          <fpage>642</fpage>
          -41335-3 21, https://doi. org/10.1007/978-3-
          <fpage>642</fpage>
          -41335-3 21
        </mixed-citation>
      </ref>
      <ref id="ref22">
        <mixed-citation>
          22.
          <string-name>
            <surname>Mebsout</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tinelli</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>Proof certi cates for SMT-based model checkers for in nite-state systems</article-title>
          . In: Piskac,
          <string-name>
            <given-names>R.</given-names>
            ,
            <surname>Talupur</surname>
          </string-name>
          , M. (eds.) Formal Methods in Computer-Aided
          <string-name>
            <surname>Design</surname>
          </string-name>
          (FMCAD
          <year>2016</year>
          ). pp.
          <volume>117</volume>
          {
          <fpage>124</fpage>
          .
          <string-name>
            <surname>IEEE</surname>
          </string-name>
          (
          <year>2016</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref23">
        <mixed-citation>
          23.
          <string-name>
            <surname>de Moura</surname>
            ,
            <given-names>L.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Bj</surname>
            <given-names>rner</given-names>
          </string-name>
          , N.:
          <article-title>Proofs and refutations, and Z3</article-title>
          . In: Rudnicki,
          <string-name>
            <given-names>P.</given-names>
            , Sutcli e, G.,
            <surname>Konev</surname>
          </string-name>
          ,
          <string-name>
            <given-names>B.</given-names>
            ,
            <surname>Schmidt</surname>
          </string-name>
          ,
          <string-name>
            <given-names>R.A.</given-names>
            ,
            <surname>Schulz</surname>
          </string-name>
          , S. (eds.)
          <source>Proceedings of the LPAR 2008 Workshops Knowledge Exchange: Automated Provers and Proof Assistants. CEUR Workshop Proceedings</source>
          , vol.
          <volume>418</volume>
          .
          <string-name>
            <surname>CEUR-WS.org</surname>
          </string-name>
          (
          <year>2008</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref24">
        <mixed-citation>
          24.
          <string-name>
            <surname>de Moura</surname>
            ,
            <given-names>L.M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Kong</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Avigad</surname>
          </string-name>
          , J., van
          <string-name>
            <surname>Doorn</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>von Raumer</surname>
            ,
            <given-names>J.:</given-names>
          </string-name>
          <article-title>The lean theorem prover (system description)</article-title>
          . In: Felty,
          <string-name>
            <given-names>A.P.</given-names>
            ,
            <surname>Middeldorp</surname>
          </string-name>
          ,
          <string-name>
            <surname>A</surname>
          </string-name>
          . (eds.)
          <source>Proceedings of the 25th International Conference on Automated Deduction, CADE-25. Lecture Notes in Computer Science</source>
          , vol.
          <volume>9195</volume>
          , pp.
          <volume>378</volume>
          {
          <fpage>388</fpage>
          . Springer (
          <year>2015</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref25">
        <mixed-citation>
          25.
          <string-name>
            <surname>Ozdemir</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Niemetz</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Preiner</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Zohar</surname>
            ,
            <given-names>Y.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Barrett</surname>
            ,
            <given-names>C.W.</given-names>
          </string-name>
          :
          <article-title>Drat-based bitvector proofs in CVC4</article-title>
          . In: Janota,
          <string-name>
            <given-names>M.</given-names>
            ,
            <surname>Lynce</surname>
          </string-name>
          , I. (eds.)
          <source>Proceedings of the 22nd International Conference on Theory and Applications of Satis ability Testing</source>
          ,
          <source>SAT 2019. Lecture Notes in Computer Science</source>
          , vol.
          <volume>11628</volume>
          , pp.
          <volume>298</volume>
          {
          <fpage>305</fpage>
          . Springer (
          <year>2019</year>
          ). https://doi.org/10.1007/978-3-
          <fpage>030</fpage>
          -24258-9 21
        </mixed-citation>
      </ref>
      <ref id="ref26">
        <mixed-citation>
          26.
          <string-name>
            <surname>Parsia</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Matentzoglu</surname>
            ,
            <given-names>N.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Goncalves</surname>
            ,
            <given-names>R.S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Glimm</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Steigmiller</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          :
          <article-title>The OWL Reasoner Evaluation (ORE) 2015 competition report</article-title>
          .
          <source>J. Autom. Reasoning</source>
          <volume>59</volume>
          (
          <issue>4</issue>
          ),
          <volume>455</volume>
          {
          <fpage>482</fpage>
          (
          <year>2017</year>
          ). https://doi.org/10.1007/s10817-017-9406-8
        </mixed-citation>
      </ref>
      <ref id="ref27">
        <mixed-citation>
          27.
          <string-name>
            <surname>Pfenning</surname>
            ,
            <given-names>F.</given-names>
          </string-name>
          :
          <article-title>Elf: A language for logic de nition and veri ed meta-programming</article-title>
          .
          <source>In: Proceedings of the 4th IEEE Symposium on Logic in Computer Science</source>
          . pp.
          <volume>313</volume>
          {
          <issue>322</issue>
          (
          <year>1989</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref28">
        <mixed-citation>
          28.
          <string-name>
            <surname>Reger</surname>
          </string-name>
          , G.:
          <article-title>Better proof output for Vampire</article-title>
          . In: Kovacs,
          <string-name>
            <given-names>L.</given-names>
            ,
            <surname>Voronkov</surname>
          </string-name>
          ,
          <string-name>
            <surname>A</surname>
          </string-name>
          . (eds.)
          <source>Proceedings of the 3rd Vampire Workshop</source>
          . EPiC Series in Computing, vol.
          <volume>44</volume>
          , pp.
          <volume>46</volume>
          {
          <fpage>60</fpage>
          .
          <string-name>
            <surname>EasyChair</surname>
          </string-name>
          (
          <year>2017</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref29">
        <mixed-citation>
          29.
          <string-name>
            <surname>Reger</surname>
            ,
            <given-names>G.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Suda</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          :
          <article-title>Checkable proofs for rst-order theorem proving</article-title>
          . In: Reger,
          <string-name>
            <given-names>G.</given-names>
            ,
            <surname>Traytel</surname>
          </string-name>
          ,
          <string-name>
            <surname>D. (eds.) ARCADE</surname>
          </string-name>
          <year>2017</year>
          .
          <source>1st International Workshop on Automated Reasoning: Challenges</source>
          , Applications, Directions, Exemplary Achievements. EPiC Series in Computing, vol.
          <volume>51</volume>
          , pp.
          <volume>55</volume>
          {
          <fpage>63</fpage>
          .
          <string-name>
            <surname>EasyChair</surname>
          </string-name>
          (
          <year>2017</year>
          ). https://doi.org/10.29007/s6d1
        </mixed-citation>
      </ref>
      <ref id="ref30">
        <mixed-citation>
          30.
          <string-name>
            <surname>Steigmiller</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Liebig</surname>
            ,
            <given-names>T.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Glimm</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          :
          <article-title>Konclude: System description</article-title>
          .
          <source>J. Web Semant</source>
          .
          <fpage>27</fpage>
          -
          <issue>28</issue>
          ,
          <issue>78</issue>
          {
          <fpage>85</fpage>
          (
          <year>2014</year>
          ). https://doi.org/10.1016/j.websem.
          <year>2014</year>
          .
          <volume>06</volume>
          .003, https: //doi.org/10.1016/j.websem.
          <year>2014</year>
          .
          <volume>06</volume>
          .003
        </mixed-citation>
      </ref>
      <ref id="ref31">
        <mixed-citation>
          31.
          <string-name>
            <surname>Stump</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Oe</surname>
            ,
            <given-names>D.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Reynolds</surname>
            ,
            <given-names>A.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Hadarean</surname>
            ,
            <given-names>L.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Tinelli</surname>
            ,
            <given-names>C.</given-names>
          </string-name>
          :
          <article-title>SMT proof checking using a logical framework</article-title>
          .
          <source>Formal Methods in System Design</source>
          <volume>42</volume>
          (
          <issue>1</issue>
          ),
          <volume>91</volume>
          {
          <fpage>118</fpage>
          (
          <year>2013</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref32">
        <mixed-citation>
          32. Sutcli e, G.:
          <article-title>The 9th IJCAR automated theorem proving system competition { CASC-J9</article-title>
          .
          <source>AI</source>
          Communications
          <volume>31</volume>
          (
          <issue>6</issue>
          ),
          <volume>495</volume>
          {
          <fpage>507</fpage>
          (
          <year>2018</year>
          )
        </mixed-citation>
      </ref>
      <ref id="ref33">
        <mixed-citation>
          33. Sutcli e, G.,
          <string-name>
            <surname>Schulz</surname>
            ,
            <given-names>S.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Claessen</surname>
            ,
            <given-names>K.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Gelder</surname>
            ,
            <given-names>A.V.</given-names>
          </string-name>
          :
          <article-title>Using the TPTP language for writing derivations and nite interpretations</article-title>
          . In: Furbach,
          <string-name>
            <given-names>U.</given-names>
            ,
            <surname>Shankar</surname>
          </string-name>
          , N. (eds.)
          <source>Proceedings of the Third International Joint Conference on Automated Reasoning, IJCAR 2006. Lecture Notes in Computer Science</source>
          , vol.
          <volume>4130</volume>
          , pp.
          <volume>67</volume>
          {
          <fpage>81</fpage>
          . Springer (
          <year>2006</year>
          ). https://doi.org/10.1007/11814771 7
        </mixed-citation>
      </ref>
      <ref id="ref34">
        <mixed-citation>
          34. development team, T.C.:
          <article-title>The coq proof assistant reference manual version 8</article-title>
          .9 (
          <issue>2019</issue>
          ), https://coq.inria.fr/distrib/current/refman/
        </mixed-citation>
      </ref>
      <ref id="ref35">
        <mixed-citation>
          35.
          <string-name>
            <surname>Vlasenko</surname>
            ,
            <given-names>J.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Daryalal</surname>
            ,
            <given-names>M.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Haarslev</surname>
            ,
            <given-names>V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Jaumard</surname>
            ,
            <given-names>B.</given-names>
          </string-name>
          :
          <article-title>A saturation-based algebraic reasoner for ELQ</article-title>
          . In: Fontaine,
          <string-name>
            <given-names>P.</given-names>
            ,
            <surname>Schulz</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            ,
            <surname>Urban</surname>
          </string-name>
          ,
          <string-name>
            <surname>J</surname>
          </string-name>
          . (eds.)
          <source>Proceedings of the 5th Workshop on Practical Aspects of Automated Reasoning co-located with International Joint Conference on Automated Reasoning (IJCAR</source>
          <year>2016</year>
          ), Coimbra, Portugal,
          <year>July 2nd</year>
          ,
          <year>2016</year>
          .
          <source>CEUR Workshop Proceedings</source>
          , vol.
          <volume>1635</volume>
          , pp.
          <volume>110</volume>
          {
          <fpage>124</fpage>
          .
          <string-name>
            <surname>CEUR-WS.org</surname>
          </string-name>
          (
          <year>2016</year>
          ), http://ceur-ws.
          <source>org/</source>
          Vol-
          <volume>1635</volume>
          /paper-10.
          <article-title>pdf (! p1 (holds (sub c0 (ex r1 c1))) (! p2 (holds (sub c1 (ex r2 c2))) (! sc (^ (concat r_1 r_2</article-title>
          ) r)
        </mixed-citation>
      </ref>
      <ref id="ref36">
        <mixed-citation>
          <string-name>
            <surname>; -------------------------------</surname>
          </string-name>
          -
          <article-title>- (holds (sub c0 (ex r c2))) (! p1 (holds (sub c1 c2)) (! p2 (holds (sub c2 c3))</article-title>
        </mixed-citation>
      </ref>
      <ref id="ref37">
        <mixed-citation>
          <string-name>
            <surname>; -----------------------</surname>
          </string-name>
          -- (
          <source>holds (sub c1 c3))</source>
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>