Ontology verbalization techniques have been introduced to automatically translate description logic (DL) axioms and derivations to natural-language texts. This way, non-expert users can be o ered explanations for subsumptions derived by systems using ontologies for knowledge-representation. We address the question of the readability and understandability of explanations generated from longer chains of inference steps, as they occur with non-trivial ontologies. An experimental design is presented to assess readers' understanding, the readability and the quality of the generated texts. The experiment tests verbalizations of derivations of di erent lengths, and assesses the e ect of a strategy proposed to shorten explanations while retaining an adequate level of informativeness.
Ontologies serve to organize concepts, terminology and relationships in a domain of interest, such as biology or medicine. Furthermore, logical consequences of this knowledge can be derived using automated ontology reasoners. To make this knowledge available to users that are not familiar with the employed formalisms (for instance, ontology languages such as OWL), verbalization techniques have been developed to automatically translate axioms and derivations to naturallanguage statements and explanations.
Example 1. As the running example throughout this paper, consider the subsumption that can be derived from the following axioms:
(PathologicalCondition u
(PathologicalCondition u The verbalization approach presented in this paper constructs a step-wise argument for this derivation in natural language, in this case:
An esophageal pathology is de ned as a pathological condition that is located in the esophagus. The esophagus is a part of the gastrointestinal tract, thus an esophageal pathology is located in a part of the gastrointestinal tract. Furthermore, since an esophageal pathology is a pathological condition, an esophageal pathology is a pathological condition that is located in a part of the gastrointestinal tract. A digestive system pathology is de ned as a pathological condition that is located in a part of the gastrointestinal tract. Thus, an esophageal pathology is a digestive system pathology.
Such an explanation combines the information from the relevant axioms in a step-wise fashion and does not require readers to be familiar with the syntax of ontology languages or description logics. However, depending on the granularity at which knowledge is modeled in an ontology, the possible derivations, and consequently the generated explanations for these derivations, can grow very long. The state-of-the-art in ontology verbalization has so far concentrated on short inference problems (one or two inference steps), but not addressed the problems that arise when verbalizing more complex derivations (more than two inference steps). Ontologies do not need to be very expressive for long and su ciently complex derivations to occur. All the derivations considered in the following remain within the OWL2 EL pro le.
This paper is organized as follows. Building on a short introduction to the considered language fragment in Section 2, we present related work on verbalization in Section 3. In Section 4 we present our own approach, which extends previous approaches by focusing on the aspect of conciseness of the generated explanations. Section 5 presents an experiment that compares explanations generated by our setup in two di erent variants, leading to the conclusions in Section 6. 2
This work remains within the DL fragment E L with some extensions that are common features of the OWL2 EL pro le (which is based on the DL E L++). As usual, class names are denoted with capital letters A; B; C; :::, role names with small letters r; s; :::, individuals with small letters a; b::: and the universal concept with >. Complex class expressions are formed by using conjunction (C1 u C2) and existential restriction (9r:C). Axioms that specify the subclass relationship between two class expressions C1 and C2, also known as subsumptions, are denoted as C1 v C2. Besides these pure E L constructors, we consider further constructors that are common in the OWL2 EL pro le. This includes the unsatis able concept ?, nominals fag which are concepts consisting of a single individual a, domain axioms dom(r; C) that are a shorthand for 9r:> v C, equivalences between concepts (mutual subclass relationship), denoted as C1 C2, disjointness axioms specifying that two class expressions C1 and C2 are disjoint, as disj(C1; C2), and role inclusion axioms r v s. We further include role inclusions that use role composition r1 ::: rk v s (called property chains in OWL2 EL).
A more comprehensive introduction to EL++ is provided by Baader et al. [
A correspondence between formal expressions in ontologies and natural-language
has been proposed in the form of controlled languages (cf. [
! (c) Every A is a C.
The generated explanation combines such patterns using the text pattern
\Statement (c) is implied because (a) ... and (b) ...". Thus, the structure
remains quite close to how proofs are presented, but the formulae are replaced
by more commonly-understandable text patterns. To test the understandability
of these patterns, an experiment was conducted where the acceptance of these
patterns was tested. Di erent rules and corresponding text patterns were found
to vary greatly in whether they were accepted as correct by experiment
participants (cf. [
The generation of explanations is based on two main components, a consequencebased reasoning system for generating step-wise derivations and a natural-language generation component to transform these formal derivations to text. In the following, these two components are introduced brie y. Based on this, we address the problem of the inconciseness of some of the generated explanations by introducing techniques and heuristics that shorten the explanations. The presented approach has been implemented as a prototype system and is available as a plugin5 for the ontology editor Protege.6 4.1
Reasoning
Derivations are constructed using a rule-based inference system with a custom
set of inference rules. Using a custom system allows us to include inference rules
that are logically redundant, but which help to obtain shorter derivations, and
thus shorter explanations. The current implementation includes and modi es
rules proposed by Nguyen et al. [
Since the current implementation of the consequence-based reasoning
procedure is not as performant on large ontologies as well-optimized tableau-based
reasoners, proof search is not performed on an entire ontology. Rather, in a
preprocessing step, a justi cation [
EP Ru+/R5
PC u 9loc.E EP v PC
R v
R EP PC u 9loc.E
EP v 9loc.E E v GTP R9/R15
EP v 9loc.GTP EP v PC u 9loc.GTP DSP
EP v DSP
PC u 9loc.GTP 5https://verbalizer.github.io/ 6http://protege.stanford.edu/ 7http://owl.man.ac.uk/factplusplus/ 8http://www.hermit-reasoner.com/
C1 C2
R1 C1 v C2 |: According to its definition, v(C1 v C2).
Ru =R2
C1 v C2 u ::: u Cn 2 i n
C1 v Ci |: Hence, v(C1 v Ci).
Ru+/R5 (1) C1 v C2 :::
(n) C1 v Cn+1
C1 v C2 u ::: u Cn+1 (1) ... (n): Since v(C1 v C2) and ... and v(C1 v Cn+1), v(C1 v C2 u ::: u Cn+1). (i), (j),...: Furthermore, since v(C1 v Ci) and v(C1 v Cj) and ...,
v(C1 v C2 u ::: u Cn+1). | : Therefore v(C1 v C2 u ::: u Cn+1).
Rv /R12 (1) C1 v C2
(2) C2 v C3
C1 v C3 (1)&(2): Since v(C1 v C2) and v(C2 v C3) it follows that v(C1 v C3). (1): v(C1 v C2), therefore being v(C3). (2): Given that v(C2 v C3), v(C1 v C3). | : Thus, we have established that v(C1 v C3).
R9/R15 (1) C1 v 9r:C2
(2) C2 v C3
C1 v 9r:C3 (1)&(2): v(C1 v 9r:C2) which [is] v(C3). Therefore, v(C1 v 9r:C3). (1): v(C1 v 9r:C2), thus v(C1 v 9r:C3). (2): v(C2 v C3), thus v(C1 v 9r:C3).
| : Therefore, v(C1 v 9r:C3). (1)&(2): v(C2) is defined as v(C3). Thus, v(C1 v C3).
| : Thus, v(C1 v C3) according to the definition of v(C2).
Rv R (1) C1 v C2 (2) C2 C3
C1 v C3 C1 C2 u ::: u Cn 2 i n
C1 v Ci | : v(C1) is defined as v(Ci u ::: u Cn). 4.2
Verbalization First, the inference steps are ordered in a linear sequence for being output as text. For this, a post-order traversal of the proof tree (as seen from the root of the tree, which contains the conclusion of the derivation) is performed. For each inference rule it is speci ed in which order its children (i.e., premises) are being output, which corresponds to the order in which the rules are indicated in Fig. 1. Then, the text patterns in Fig. 1 are applied to transform the derivation into text. For each rule, the exact pattern that is applied depends on whether one or several of its premises have been presented immediately before in the generated text (for example, as a conclusion of a previous step), in which case they should not be repeated.
In the running example, the rst step to be output is the application of R (top left in the proof tree), with its template producing by default: \An esophageal pathology is de ned as a pathological condition that is located in the esophagus". The second application of R (top center in the proof tree) produces no output, for being detected as identical to the previous output. The next rule application to be output is R9/R15. Since the rst premise is counted as being \covered" by the previous output, only the second premise (marked as (2)) is output, together with the conclusion, i.e. the second pattern is chosen: \The esophagus is a part of the gastrointestinal tract, thus an esophageal pathology is located in a part of the gastrointestinal tract." As can be seen in this example, some intermediate conclusions remain implicit (e.g. EP v PC and EP v 9loc.E as conclusions of R ), a strategy we discuss below. 4.3
Techniques and Heuristics for Improving Text Quality The aim of conciseness of the verbalized derivations is pursued at three levels. At the level of the generated proofs, inference rules are used that correspond to two or more applications of simpler rules and provide a kind of shortcut. In the running example, one application of R v (the last step) replaces the application of R1 and R v /R12.
Secondly, some inference rules are considered to be trivial and their application is simply omitted altogether in the text output. Among the rules in Fig. 1, Ru =R2 represents such an inference rule. Furthermore, for some of the extra inference rules, the conclusion is not being output (again, for being considered obvious). As illustrated above, this is the case for R .
Finally, at the level of individual statements, the text patterns are designed
such that unnecessary repetitions are avoided. For example, the \middle term"
in R v /R12 only needs to be mentioned once (in contrast to the pattern used
by [
When using more \complex" inference rules and hiding inference rule applications (referred to as \shortening" in the following), the question is whether the understandability of the resulting explanations is retained. This prompted an investigation presented in the following section. 5
To test the understandability and text quality of generated explanations for derivations in ontologies, a questionnaire-based experiment was devised. This experiment allowed for a comparison between explanations in their unshortened form and their shortened form according to the presented heuristics. Since no such experiment has been conducted before, we explored with a small number of participants whether its design is suited as an instrument for assessing di erences between shortened and unshortened explanations. We consider the results informative as a preparation for larger studies and also for the further development of the presented verbalization techniques.
Procedure Participants were randomly assigned to two groups. Eight explanations were shown to each participant. The rst group received four explanations in their unshortened version and four in their shortened version. The second group received the corresponding shortened and unshortened alternatives of these explanations. The shortened version of an explanation uses logically redundant rules (R v and R in the running example), whereas the unshortened version uses only the most basic rules. In the unshortened case, verbalizations of Ru =R2 are omitted. For comparison, Fig. 2 shows both versions for the running example used in the experiment.
As an objective test for participants' careful reading and understanding, participants were asked to indicate for each explanation whether it is logically correct. Two out of the eight explanations were manipulated to be erroneous by replacing one occurrence of a classname by a di erent one which was not part of the initial axioms. This manipulation was designed to ensure that participants read the text properly, but not to test their formal reasoning skills.9 Note also that the two last sentences of Fig. 2 (b) are generated from one rule application of R v . The employed pattern was di erent from the one presented in Fig. 1, in that it lacks the part \de ned as" to make clear that it refers to an equivalence and not a subsumption. This de ciency was detected during the experiment and corrected in the verbalization system.
Understandability and readability of the explanations were assessed using a
questionnaire to be answered on a 7-point scale. Participants were asked for
ratings pertaining to key aspects of the text quality of the presented explanations;
namely understanding/comprehension, conciseness and appreciation. The topic
domains for which the explanations were generated were chosen to be relatively
9It is well known that untrained participants in reasoning experiments do not always
apply classical logical reasoning. For example, consider the Wason selection task [
Additionally, an esophageal pathology is located in the esophagus.
The esophagus is a part of the gastrointestinal tract, thus an esophageal pathology is located in a part of the gastrointestinal tract.
Furthermore, since an esophageal pathology is a pathological condition, an esophageal pathology is a pathological condition that is located in a part of the gastrointestinal tract.
According to the de nition of a digestive system pathology, a pathological condition that is located in a part of the gastrointestinal tract is a digestive system pathology.
Thus, we have established that an esophageal pathology is a digestive system pathology.
(a) Unshortened Explanation An esophageal pathology is de ned as a pathological condition that is located in the esophagus. The esophagus is a part of the gastrointestinal tract, thus an esophageal pathology is located in a part of the gastrointestinal tract.
Furthermore, since an esophageal pathology is a pathological condition, an esophageal pathology is a pathological condition that is located in a part of the gastrointestinal tract.
A digestive system pathology is a pathological condition that is located in a part of the gastrointestinal tract. Thus, an esophageal pathology is a digestive system pathology.
(b) Shortened explanation employed in the experiment in unshortened and shortened form. abstract and unfamiliar to most participants. Therefore, when judging readability one has to take into account that the domain itself may be challenging, an aspect for which questions were included under appreciation. The items are:
{ I can follow the reasoning steps presented in the explanation. (Question 1) skipped. (Question 2) (Question 3) { I
nd that some steps in the explanation are so obvious that they should be { The explanation conveys less information than I need to fully understand it. { I
nd that the explanation should be made more concise. (Question 4) Appreciation
(Question 5) { The text of the explanation is well-formed (according to writing conventions) { The sentences are arranged such that they t together well. (Question 6) { I
nd the text easy to read. (Question 7) { I am familiar with the technical terms in this text. (Question 8) { The technical terms make it di cult for me to follow the text. (Question 9) { The topic of the text makes it di cult for me to read the text. (Question 10)
At the beginning of the experiment, participants were informed that they will be asked to provide judgments for automatically generated explanations. An example was shown together with the correct answer and an explanation (reproduced in [17, Appendix B]). Then participants were asked for demographic data and prior experience with the following elds of science: computer science, arti cial intelligence, mathematics/formal logic, philosophy, linguistics, physics, biology, medicine, chemistry. Participants received the eight explanations in random order, each with the same set of questions. The presentation of each explanation was preceded by a presentation of the axioms that were assumed to hold (in verbalized form) and the conclusion derived from them (also verbalized). A screenshot showing the running example together with the associated questionnaire is reproduced in [17, Appendix B]. After the experiment, participants were o ered a free-text eld for any comments on the explanations and were thanked for their participation. The questionnaire was administered using LimeSurvey.10 Participants Seven current and former members of Ulm University took part in the experiment. None of them was involved in the development of the presented verbalization techniques and the experiment. Participants included one female and six males and were aged between 20 and 34. Four indicated to be uent in English, two indicated good English pro ciency and one intermediate pro ciency. All participants reported experience in computer science, four in Arti cial Intelligence, and three in mathematics/formal logics.
Materials To generate a pool of verbalized derivations, the verbalization tool was run on the ontologies in the TONES repository.11 To enable the retrieval of subcorpora of verbalizations according to a set of criteria (e.g. number of inference steps, employed rules), explanations and their properties were stored in a MySQL12 database. The TONES ontology repository was chosen for including a good level of diversity regarding the included ontologies' domains and complexities. Since TONES includes ontologies of di erent levels of expressivity, only derivations that fell into the language fragment handled by the verbalization tool were generated. Still, a su ciently large and diverse corpus of explanations in various domains (anatomy, chemistry, geology, physics, but also the \pizza" tutorial domain13) was obtained. Most importantly, this set represents \ordinary" ontologies that were not designed with verbalization in mind, but which are likely to be encountered by potential users of the verbalization tool. To restrict the scope of the experiment in a sensible way, the pool of considered explanations was further narrowed down according to the following criteria: { The explanations are of medium length (3{5 inference steps when shortened). { The verbalizations make use of additional rules or the skipping of rules in the presentation, as discussed in this paper, such that the e ect of shortening can be studied. { The axioms from which the verbalizations are generated are plausible according to common sense. Some of the included ontologies were found to contain unusual or erroneous formalizations of their domain. For example, some ontologies include axioms for a concept { which in the real world is known to have instances { such that the concept becomes unsatis able (probably 10https://www.limesurvey.org/de/ 12https://www.mysql.com/de/ 13http://mowl-power.cs.man.ac.uk/protegeowltutorial/resources/ ProtegeOWLTutorialP4_v1_3.pdf long explanation short explanation 1 2 3 4 7 8 9
10 5 Question 6
unintendedly) and therefore the concept can be shown to be a subconcept of any concept. Some ontologies also contain equivalences with apparently unintended consequences. { Verbalizations that rely on equivalences between several concepts with the same classname in di erent ontologies (with di erent URLs) were excluded.
Such equivalences result in statements such as \since a person is a person...". { Explanations with excessively long concept names were also excluded. However, since long concept names are quite common in the investigated ontologies, compound concept names of up to three words were accepted. In order to provide a realistic selection of explanations for the experiment (instead of hand-picking some \nice" explanations), and thus a realistic evaluation of the verbalization tool, the presented explanations were selected at random from the pool that ful lled the criteria stipulated above. By default, the verbalization tool provides a simple highlighting of concept descriptions by displaying them in a di erent color (blue) than the surrounding text (black). Without such highlighting, the reading of the often long sentences containing long compound concept names is made unnecessarily tedious.
Results The two manipulated explanations were always correctly detected to be wrong by the participants except in one case. The non-manipulated explanations were mostly judged as correct, as predicted. One participant noticed the problem with the verbalization of R v .
Figure 3 shows the subjects' averaged scores on Questions 1-10 for long and short explanations. The understandability of the explanations was judged favorably by the participants (Question 1), but not always found to be ideal. Participants' opinions on conciseness (Questions 2-4) turned out to be mixed and not too strong. Text quality (Questions 5-10) was also judged favorably. The answers of the participants to most questions are concentrated within relatively precise ranges and do not occupy the endpoints of the scale. Inter-rater reliability was assessed using Kendall's coe cient of concordance (corrected for ties) for the seven subjects' averaged scores on Questions 1-10 for long and short explanations (as shown in Fig. 3) and was found to be good (W = 0:617, 2(19) = 82:1, p < 0:001). Concordance for individual answers measured separately for both experimental groups (with three and four participants each) was also good (W = 0:759, 2(79) = 180, p < 0:001 and W = 0:561, 2(79) = 177, p < 0:001).
This is a favorable outcome with respect to the question whether this experimental setup can be used as an instrument to detect di erences in the scores related to the experimental manipulation, i.e. the shortening of the explanations, provided a larger number of participants to ascertain su cient statistical power. For example, if a larger number of participants leads to a statistically signi cant di erence in responses to Question 4 (the question whether the explanation should be made more concise), this would provide evidence for a positive e ect of shortening on perceived conciseness of the explanations. 6
This work has, for the rst time, investigated the generation of verbalized explanations for non-trivial derivations consisting of several inference steps. In the evaluation, verbalizations for up to seven inference steps (e.g. the running example in its unshortened form) were considered. To make such explanations more concise, we propose the use of an extended set of inference rules and the hiding of inference steps in the presentation. The presented experimental design can be used to test whether such adjustments impact the understandability and quality of the generated explanations. Our small-sample study provides a rst indication of how these measures turn out, which will be helpful for conducting a larger follow-up study. Whereas the TONES repository was considered an instrumental choice for generating explanation material in a rst step, more up-to-date corpora of ontologies should be considered in future studies. Furthermore, the experiment provided some rst evidence supporting the understandability of such explanations in general, in spite of the considerable length of the derivations and the technical jargon used in the considered domains. In how far this is also the case for users with less background in technical domains than the participants of this study (in this case, all had a background in computer science) is to be investigated as part of future work.
Acknowledgments. We acknowledge the support of the Transregional Collaborative Research Center SFB/TRR 62 \A Companion-Technology for Cognitive Technical Systems" funded by the German Research Foundation (DFG). We thank the three helpful reviewers and all experiment participants.