=Paper=
{{Paper
|id=Vol-2931/ICBO_2019_paper_20
|storemode=property
|title=Wordified Ontologies: Evaluating a Novel Paradigm for Ontology Editing
|pdfUrl=https://ceur-ws.org/Vol-2931/ICBO_2019_paper_20.pdf
|volume=Vol-2931
|authors=Aisha Blfgeh,Phillip Lord
|dblpUrl=https://dblp.org/rec/conf/icbo/BlfgehL19
}}
==Wordified Ontologies: Evaluating a Novel Paradigm for Ontology Editing==
https://ceur-ws.org/Vol-2931/ICBO_2019_paper_20.pdf
Wordified Ontologies: Evaluating a Novel Paradigm for Ontology Editing
Aisha Blfgeh1,2 and Phillip Lord1
1
School of Computing, Newcastle University, UK
2
College of Computer Sciences and Engineering, University of Jeddah, Saudi Arabia
Abstract User evaluation is a standard part of the software engineering
cycle; it has previously been applied to various aspects of ontology
engineering, including the use of foundational ontology in ontology
Ontologies can be edited with tools such as Protégé, or with development [7], and finding frequent user activities in Protégé
various other forms of source code. The editing process in- using Eye-tracking analysis [8].
volves insertion, deletion, or fixing errors. In this paper we
propose an editing process using Microsoft Word, where users Others have concluded that [9] the tools used for reading and
can manipulate any text, adding comments and use all the fa- understanding an ontology play a critical role in determining the
usability of that ontology. Furthermore, by using a verbalised
cilities provided by a familiar word-processing environment.
version of the ontology in their evaluation practice, they found
This new technique for visualising and editing ontologies has that this supports the identification of mistakes in the ontology.
been tested and evaluated; the results are promising as mod-
ifying an ontology in Word is preferred by users to Protégé In our previous work, we have shown that it is possible to wordify
for some ontology editing tasks. We suggest, therefore, that an ontology; however, to demonstrate that it is also useful to do
alternative text based representations and office tools may be this, we need some form of evaluation. In this study, we assess the
useful in the ontology engineering lifecycle. comprehension/manipulation of an ontology presented in this way.
We conducted several experiments where users read and manipu-
Keywords: lated the Wordified ontology as well as performing the same tasks
Ontology Editing, Wordified ontology, User Evaluation using Protégé; then we assessed their performance and measured
their level of satisfaction using feedback forms.
This paper is organised as follows: first we describe existing
Introduction alternative tools for ontological documentation, then we show our
new visualisation version of the ontology. After that, we describe
the experiments with users and show the results. Finally, we
Ontology development is a collaborative process which involves a discuss the results and draw conclusions.
sustained interaction between domain specialists and ontology de-
velopers. As shown in Figure 1, we designed a document-centric
workflow to enable the co-ordinated use of Microsoft Office tools
(Excel and Word) for ontology development. An Excel spread-
sheet is used as a source of values that instantiate patterns, de-
fined in Tawny-OWL 1 source code [1], to construct the ontology.
We have also generated a Word document of an ontology; we de-
note this representation as a Wordified Ontology. It allows domain
specialists to cooperate and interact with the developers in edit-
ing the ontology during the development process [2]. The use of
Word documents enables us to include the documentation of the
ontology with the computational components.
Figure 1: Document-Centric Ontology Development Work-
In some ways, this is similar to an “Intermediate representation” flow
as defined by Rector et al˙ [3], where the knowledge from experts
is transformed into a semi-formal syntax that the ontology devel-
opers use to deal with ontological information. Also, using dif-
ferent syntaxes to instantiate patterns is not new; for example in Alternative representations of Onto-
OPPL (Ontology Pre-Processing Language) [4], and DOSDP [5]
where an abstract syntax is used for effectively editing the on- logical Knowledge
tology. Office tooling has been integrated into the ontology de-
velopment process before such as with Populous [6] and our own,
There have been many other attempts to present ontological
Excel-based approach [1]), however only with the more structured
knowledge in predominately textual formats. For example, OWL-
forms of spreadsheets. To our knowledge, the use of arbitrary
Doc is a Protégé plugin that generates HTML documentation [10].
syntax tightly integrated and presented in rich Word documents
It was inspired by JavaDoc, which does something similar with
is novel.
Java source code. The aim of OWLDoc is to provide a browsable,
1 https://github.com/phillord/tawny-owl but not editable, experience of the ontology inside Protégé or in
Copyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
1
a standalone web browser. While this works well, if errors are how to build a “Pizza Ontology”. We have included an explana-
discovered the user has to move into a different environment to tion of logical classifications of the ontology in a narrative style
fix them. as well as the Tawny-OWL source code which fulfils the chrono-
logical story of Pizza Ontology construction. Unlike Protégé, this
Another mechanism for visualisation 2 was the “Intermediate Rep- narrative structure of the ontology offers the ability to explore
resentation” [3]. This mechanism produces semi-structured text ontologies as a linear, narrative document.
representations (Figure 2a), which were designed for domain spe-
cialists to read and edit, before being checked and cleaned by A Wordified ontology is the narrative version of the ontology and
knowledge engineers who would correct their syntax and seman- includes the whole source code of an ontology using Tawny-OWL
tics; this representation was used to enable authoring of the final syntax. We intentionally include the Tawny-OWL source code be-
ontology. Many years after, a similar practice has been incorpo- cause of its textual representation. It was designed after Manch-
rated into software engineering with Behaviour-Driven develop- ester syntax to be straightforward allowing a developer (or non de-
ment tools such as Cucumber; this allows semi-structured state- veloper) to read it without deep knowledge of the syntax. There-
ments (Figure 2b) to define requirements which can be tested fore, we chose to have the complete source code of the ontology in
computationally as part of the software testing lifecycle. the Wordified ontology, also we have the other purpose of testing
the ability to comprehend the Tawny-OWL source code by users.
MAIN plastic construction
ACTS_ON ulna
BY_TECHNIQUE transplanting
ACTS_ON bone
WITH immobilising
BY_MEANS_OF fixation device
(a) Intermediate representation
Feature: Return Microwave
Scenario: Fred gets his money back
Given Fred has bought a microwave
And the microwave cost 100
When we refund the microwave
Then Fred should be refunded 100
(b) Cucumber
Figure 2: Intermediate representation and Cucumber text
Figure 3: Wordified ontology: A structured and readable text
with headings and source code syntax highlighted
An alternative to an intermediate representation is to use an on-
tology syntax, which is designed to be readable to knowledge en-
gineers such as Manchester Syntax ; this was aimed at editing and Therefore, our mechanism for adding ontology documentation is
representing all the aspects of an ontology, including the extra- based on adding a rich commentary text within the Tawny-OWL
logical parts which are critical for understanding the ontology in source code through out the ontological and software statements:
the context of its domain [11]. Unlike other syntax it is frame- a form of literate programming. Additionally, we use markup an-
based, grouping statements about a single domain concept, rather notations to produce a structured documentation as well as the
than axiom-based where statements are essentially unordered. Tawny-OWL source code text. This form of source code can be
transformed into a Word document; in Figure 3, we show how
We use the mechanism of adding comments in the source code; we the text is structured and formatted with headings and subhead-
have previously described these as literate ontologies [12, 13] where ings; the source code of Tawny-OWL is also shown in a syntax
we produce LaTeX, ASCII and OWL versions from a single source highlighted text blocks.
code. In our case, we produce a Microsoft Word version, and the
final Wordified ontology is formatted as we desired. Figure 3 shows Next, we describe the evaluation process of Wordified ontologies.
an extract of an Wordified ontology. In the following section we
explain the mechanism for generating an Wordified ontology.
Evaluation Experiments
Microsoft Word and Ontology Repre-
The aim of our evaluation is to discover how easy for users to
sentation comprehend and interact with such a new form of the ontology.
We achieve this by having the users read an ontology, understand
their structure and search for errors introduced into our sample
Although there is no clear structure of what ontology documen-
ontologies.
tation should be, we started by creating a guidance document on
2 We use the term visualisation in a broader fashion than is common, We arranged controlled testing sessions to test the comprehen-
to include text on screen sion of the Wordified ontology and compare the performance by
2
(a) Wordified Ontology (b) Ontology in Protégé
Figure 4: Testing ontologies
visualising the same ontology in Protégé. In other words, the par- ticipants write their feedback electronically about the following
ticipants examine the same ontology in the two different visuali- aspects:
sations. These visualisations can be seen in Figure 4 below3 . Our
participants were mostly postgraduate students and researchers
• Clarity of ontology structure.
with a variety of backgrounds and experience in ontologies. Some
have a decent knowledge in building ontologies and some are to- • Understanding the construction flow of the ontology.
tally unfamiliar. • The ease of reading the ontology.
We wished to test the ease of reading and understanding the • Editing the ontology.
Wordified ontology as well as how easy it is to discover errors. • Finding errors.
Therefore we injected a few errors, both logical and extra-logical,
into the ontology for users to find during the test. We prepared
The feedback questions aim to measure how easy it is to read,
multiple versions of ontologies, so that the participants could ex-
comprehend, and edit Wordified ontology. All answers were rated
plore different visualisations of the same ontology, but with dif-
using a five point Likert scale.
ferent sets of errors.
In each version of the ontology, we have engineered four or five
errors to be detected. These errors are either in classification or
in some logical specifications of properties/classes, such as the
range/domain of a property, and the disjointedness of sub-classes.
Figure 4 shows examples of one error in Wordified ontology and in
Protégé. Additionally, We have not included any spelling errors
because they are instantly can be detected by the Microsoft Word
software spelling checker, and in Protégé test; the participants are
not allowed to run any reasoner checks.
After introducing the experiment objectives and the tasks to per-
form in the test, we provided instruction sheets in details for more
assistance during the session. A participant starts with either
a Wordified ontology or in Protégé, the selection process being
Figure 5: Evaluation Experiment Workflows
randomly performed by the experimentor to ensure the variety
between the participants in one session 5 . Hence, there are two
different paths through testing experiment, depending on which
version of the ontology they see first (Figure 5). This was done to
ensure that preference between visualisation would not be affected
by either fatigue or use of knowledge from the first test affecting
the second. Additionally, we set an equal time limit to spend in
each part to ensure fairness of the two tests. Finally, the par-
3 Each ontology in this Figure has an error, can you find them?4
5 There was no particular procedure to randomise the selection; we
tried to maintain a reasonable distribution amongst our sample.
3
Expertise level of the participants
To maintain the fairness of our test and ensure the variety of expe-
rience levels, we asked the subjects about their level of experience
and education. We asked basic demographic data as some peo-
ple may under/over estimate their abilities depending on many
factors, such as gender, age, etc (data not shown). The subjects
experience level in ontology construction and usage is shown in
Figure 6. About 32% and 40% of the subjects are ”Somewhat
Familiar” with ontology usage and construction respectively. We
had about 16% and 20% of the participants consider themselves
to have a ”Mastery” level in ontology usage and construction. In
our experiment, therefore, the experience was reasonably spread Figure 8: Understanding the construction flow of the ontology
across range of different expertise; this allowed us to get a rea- in: Wordified ontology and Protégé
sonable sample size, which would not have been possible if we
restricted, for example, only to experts; it is probably reflective of
the ontology development community, which also has many dif-
ferent levels of expertise. None of the participants were members Editing the ontology
of our lab, and had not seen Wordified ontologies previously.
Editing the ontology in this context refers to any form of the
modification: deletion, insertion or update of the ontology. In a
Wordified ontology, users can also add comments and annotate
Evaluation results the text using the Track changes facility in Word. We asked the
participants to turn on Track changes before they perform any
In this section, we explore the results of the evaluation and de- changes in the ontology. This helps in saving time and effort for
scribe the feedback answers from users about the five aspects men- the ontology developers when updating the source code of the
tioned in the previous section showing their preferences in these ontology accordingly.
aspects.
As shown in the Figure 9 below, most participant are either “Satis-
fied” or “Strongly Satisfied” with their performance in the editing
tasks during the test. This indicates modification of a Wordified
Reading and understanding the construction ontology is preferred to modification in Protégé.
flow of the ontology
In this context, reading the ontology refers to the action of ex-
ploring and browsing the ontology. Because of our test on the
Wordified ontology, we use the term “reading” as it also has more
text and documentation of the ontology. Also, understanding the
construction flow refers to the ability to follow the development
process of the ontology regardless of the representation format.
Generally, over 60% of the participants find that the ontology is
easy to read in both representations as shown in Figure 7. Al-
though we designed the Wordified ontology with comprehensive
text that explains the construction of the ontology, 40% feel “Neu-
tral” about understanding the construction flow of the ontology
Figure 9: Results of editing the ontology in: Wordified ontol-
where the same percentage can understand the construction flow
in the Protégé (see Figure 8).
ogy and Protégé
Finding Errors
One of the participants task was to search for the errors we in-
cluded in the ontology and correct them. We intended to discover
how easy and quickly to spot errors in the ontology; hence we
limited the time available for this task. This was a hard task,
most participants 6 managed to detect at most a single error in
the Wordified ontology and two errors in Protégé.
The participants feedback results are quite similar in both parts,
looking at Figure 10, there are nearly the same number of par-
Figure 7: The ease of reading the ontology in: Wordified ticipants (nine and ten) either “Satisfied” or “Strongly Satisfied”
ontology and Protégé 6 6 participants using Wordified ontology and 4 using Protégé.
4
(a) Experience Level in Ontology Usage (b) Experience Level in Ontology Construction
Figure 6: Experience level of our subjects: a) Ontology Usage and b) Ontology Construction
with this task in Wordified ontology and Protégé. A quarter find Table 1: The preferences of user in using the Wordified on-
it difficult to perform the task of finding errors in Protégé and tology over Protégé.
nearly a third in Wordified ontology. This could be due to the
time limit we set in our experiments. Wordified Ontology Protégé Both
Reading 20% 68% 12%
Editing 56% 24% 20%
Learning 20% 56% 24%
Figure 10: Results of Finding Errors in the ontology using:
Wordified ontology and Protégé Figure 11: Users preferences in “Editing” according to the
their level of expertise
Discussion
Overall preferences In this paper, we have evaluated whether an alternative form of
representation, namely the Wordified ontology, is useful and us-
able by ontology users, both experts and non.
At the end of the session, we asked the subjects about their final
preferences in using the Wordified ontology over Protégé in the Our analysis of other work in this area shows that, in the field of
main three aspects: Reading (Exploring), Editing and Learning ontology engineering, user experience testing is relatively limited
about the ontology. The overall preferences of the users between with notable exceptions being the evaluation of an application
the two forms are shown in Table 1. Wordified ontology seems ontology [9, 7], and the analysis of Protégé activities [8]. This
to be preferable in editing, where Protégé is more convenient in paper shows the value of this form of user evaluation because the
reading and learning the ontology due to the hierarchical repre- results were substantially different from our initial expectations:
sentation. we thought people would prefer Protégé, especially for editing, as
it is more familiar and has been longer in development.
We also split the results of “Experts” preferences and “beginners”
(or non-experts), in order to ensure that the level of experience Despite that there is no significant difference between the two
does not affect users perspectives. We found no divergence in the formats of the ontology in our test (the p-values are less than 0.05),
results; the Wordified ontology remain the preferable as can be which means that the results are relatively close between both
seen in the Figure 11 below. formats; the Wordified ontology seems to compete the Protégé
5
software especially in the area of documenting and editing the Address for correspondence
ontology, this is due to the familiarity of the Microsoft Word for
different kinds of users, which requires no prior skills to deal with
these Wordified ontologies. Aisha Blfgeh
a.blfgeh1@newcastle.ac.uk
Alternative representations have been tried before such as the abelfaqeeh@kau.edu.sa
previously mentioned “Intermediate Representation” [3], also the
auto-generation of textual class definition [14]; in these cases, the Phillip Lord
representations have been textual and did not focus on the appli- phillip.lord@newcastle.ac.uk
cation that the users would use to interact with the text. Like-
wise, for more formal representations, Manchester Syntax [11] and
DOSDP [5] efforts have focused on the representation alone.
References
In this paper, we have tested the utility of Wordified ontologies,
and this shows that the Wordified ontology has a promising place [1] Aisha Blfgeh, Jennifer D. Warrender, Catharien M. U.
in the future of ontology development. Although our word-based Hilkens, and Phillip Lord. A document-centric approach
presentation of ontologies is relatively immature, users still found for developing the tolapc ontology. In Frank Loebe, Mar-
it useful for understanding and debugging ontologies. Counter to tin Boeker, Heinrich Herre, Ludger Jansen, and Daniel
this, most users preferred Protégé for understanding overall flow, Schober, editors, Proceedings of the 7th Workshop on On-
probably because of the hierarchical browser, as supported by the tologies and Data in Life Sciences, ODLS 2016, organized by
previous analysis of Vigo et al˙ [8], which showed that users spent the GI Workgroup Ontologies in Biomedicine and Life Sci-
45% of their time looking at it. ences (OBML), Halle (Saale), Germany, September 29-30,
2016., volume 1692 of CEUR Workshop Proceedings, pages
There is still significant work to be done in improving the presenta- 1–6. CEUR-WS.org, 2016. http://ceur-ws.org/Vol-1692/
tion of Wordified ontologies. While it is well understood how com- paperB.pdf.
ments can be written in a light-weight markup and transformed
[2] Aisha Blfgeh and Phillip Lord. User and developer inter-
into Word structure, we lack a good understanding of what text
action with editable and readable ontologies. In Proceed-
should go into documentary comments and what should be repre-
ings of the 8th International Conference on Biomedical On-
sented, for example, as rdfs:comments into the annotation of the
tology (ICBO 2017), Newcastle-upon-Tyne, United Kingdom,
ontology. Similarly, presenting the formal parts of the ontology as
September 13th - 15th, 2017., 2017. URL http://ceur-ws.
source code in Word is clearly not ideal; a more textual represen-
org/Vol-2137/paper_28.pdf.
tation (such as [3] or [14]) might be preferred by users. For a tool
such as Tawny-OWL, the non-ontological parts of the source code [3] A.L. Rector, P.E. Zanstra, W.D. Solomon, J.E. Rogers,
are also challenging. Finally, we need to explore different ways R. Baud, W. Ceusters, W. Claassen, J. Kirby, J.-M. Ro-
of integrating Wordified ontologies into the development process; drigues, A. Rossi Mori, E.J. Van der Haring, and J. Wagner.
either for new or existing ontologies. Reconciling users’ needs and formal requirements: issues in
developing a reusable ontology for medicine. IEEE Transac-
Tools like this have, however, proven to be very popular in soft-
tions on Information Technology in Biomedicine, 2(4):229–
ware development, and form the basis for behaviour-driven devel-
242, 1998. ISSN 10897771. doi: 10.1109/4233.737578. URL
opment (BDD) [15]; from here, we have taken some of our motiva-
http://ieeexplore.ieee.org/document/737578/.
tion. In addition, the use of Office tools remain popular for data
handling, even though they might appear to be poorly suited for [4] Mikel Egaña, Robert Stevens, and Erick Antezana. Trans-
it, because users are very familiar with them. forming the Axiomisation of Ontologies: The Ontology Pre-
Processor Language. Proceedigns of OWLED, 2009. doi:
In our tests, we have shown that, even though immature, Wordi- 10.1038/npre.2009.4006.1.
fied ontologies can stand alongside or as a partial replacement for
tools such as Protégé. With further development and close at- [5] David Osumi-Sutherland, Mlanie Courtot, James Balhoff,
tention to user requirements, we believe that they could provide and Christopher Mungall. Dead simple owl design pat-
substantial benefits when properly integrated into the Ontology terns. Journal of Biomedical Semantics, 8, 06 2017. doi:
Engineering lifecycle. 10.1186/s13326-017-0126-0.
[6] Simon Jupp, Matthew Horridge, Luigi Iannone, Julie Klein,
Stuart Owen, Joost Schanstra, Katy Wolstencroft, and
Robert Stevens. Populous: a tool for building owl ontolo-
gies from templates. BMC Bioinformatics, 13(Suppl 1):
Acknowledgement S5, 2011. doi: 10.1186/1471-2105-13-S1-S5. URL http:
//dx.doi.org/10.1186/1471-2105-13-S1-S5.
[7] C Maria Keet. The use of foundational ontologies in ontology
We would like to express our thanks to Newcastle university for development: an empirical assessment. In Extended Semantic
supporting this research. Also, many thanks to the University of Web Conference, pages 321–335. Springer, 2011.
Jeddah, Saudi Arabia for funding the scholarship.
[8] Markel Vigo, Caroline Jay, and Robert Stevens. Constructing
conceptual knowledge artefacts: activity patterns in the on-
tology authoring process. In Proceedings of the 33rd Annual
6 The property range in Wordified ontology and the ”StuffedCrust- ACM Conference on Human Factors in Computing Systems,
Base” class in Protégé. If you found them, WELL DONE! pages 3385–3394. ACM, 2015.
6
[9] He Tan, Anders Adlemo, Vladimir Tarasov, and Mats E Jo-
hansson. Evaluation of an application ontology. In Proceed-
ings of the Joint Ontology Workshops 2017 Episode 3: The
Tyrolean Autumn of Ontology Bozen-Bolzano, Italy, Septem-
ber 21–23, 2017, volume 2050. CEUR-WS, 2017.
[10] CO-ODE OWL Plugins. Owldoc. https://github.com/
co-ode-owl-plugins/owldoc, 2016.
[11] Matthew Horridge, Nick Drummond, John Goodwin, Alan
Rector, Robert Stevens, and Hai Wang. The manchester owl
syntax. 01 2006.
[12] Phillip Lord and Jennifer D. Warrender. A highly literate
approach to ontology building. abs/1512.04250, 2015. http:
//arxiv.org/abs/1512.04250.
[13] J. D. Warrender and P. Lord. The karyotype ontology: a com-
putational representation for human cytogenetic patterns.
Bio-Ontologies, 2013.
[14] Robert Stevens, James Malone, Sandra Williams, Richard
Power, and Allan Third. Automating generation of textual
class definitions from owl to english. Journal of Biomedical
Semantics, 2(Suppl 2):S5, 2011.
[15] Bdd: Learn about behavior driven development, Dec 2018.
URL https://www.agilealliance.org/glossary/bdd/#q=
~(infinite~false~filters~(postType~(~’page~’post~’
aa_book~’aa_event_session~’aa_experience_report~’
aa_glossary~’aa_research_paper~’aa_video)~tags~(~’
bdd))~searchTerm~’ ~sort~false~sortDirection~’
asc~page~1).
7