The work presented in this paper demonstrates an evaluation procedure for a real-life application ontology, coming from the avionics domain. The focus of the evaluation has speci cally been on three ontology quality features, namely usability, correctness and applicability. In the paper, the properties of the three features are explained in the context of the application domain, the methods and tools used for the evaluation of the features are presented, and the evaluation results are presented and discussed. The results indicate that the three quality features are signi cant in the evaluation of our application ontology, that the proposed methods and tools allow for the evaluation of the three quality features and that the inherent quality of the application ontology can be con rmed.
Ontologies as a modelling approach have long been recognized by the software
engineering community (e.g. [
The ontologies developed in the research project presented in this paper can be considered to be application ontologies. They contain the knowledge of a particular domain, that is the requirements speci cation for software, to be able to achieve a speci c task or support a particular application, that is the generation of software test cases. An ontology, like all engineering artefacts, needs a thorough evaluation to be able to put con dence in it and its intrinsic quality features. Not only will the quality of an ontology have a direct impact on the quality of the results originating from it, the ontology's quality during the ontology development should also be measured in a quantitative way, to decide whether the ontology will be able to meet its assigned goals or not. In this paper we demonstrate an ontology evaluation carried out in the aforementioned research project. We have focused on the evaluation of three speci c quality features, i.e. usability, correctness and applicability, and proposed methods and tools for the evaluation of the quality features.
The remainder of the paper is organised as follows. Section 2 presents one of the application ontologies developed in our research project. In section 3, we discuss the quality features used to evaluate the ontology and the methods for evaluating the features. Section 4 presents the evaluation tools and the ontology evaluation with the results. Section 5, nally, presents our conclusions and discussions on future work.
In this section, we rst present one of the application ontologies that were
developed in our research project, along with the task it supports. The task of the
ontology is to capture the knowledge contained in a set of given software
requirements in an e ort to automate the creation of software test cases. The ontology
represents the requirements of a communication software component pertaining
to an embedded system situated in an avionics system. The software component
is fully developed in compliance with the DO-178B standard [
The ontology was created using Protege. Beside the ontology itself, documentation was created to give an overview of the represented knowledge and describe the design of the ontology. The current version of the ontology contains 42 classes, 34 object properties, 13 datatype properties, and 147 instances in total. Figure 1 shows the ontology fragment for one particular functional requirement, in this case the SRSRS4YY-431, a requirement that has a focus on error handling. Ontology fragments for the remaining individual requirements related to the communication software component can be represented in a way similar to the one in Figure 1. The SRSRS4YY-431 de nes that if the communication type is out of its valid range, the initialization service shall deactivate the UART (Universal Asynchronous Receiver/Transmitter), and return the result "comTypeCfgError". In the gure, the rectangle boxes represent the concepts of the ontology; the rounded rectangle boxes represent the instances; and the dashed line rounded rectangle boxes provide the data values of the datatype property for instances.
The ontology has been designed to support inference rules to generate the
software test cases [
The challenge in ontology evaluation is to determine the quality features that
are to be evaluated, along with the evaluation method. Many di erent
evaluation features have been discussed in literature (e.g. [
In the context of this paper, we consider usability of an application ontology to be a set of attributes that describe the e ort needed by a human to make use of the ontology. The usability feature provides a measure of the quality a user experiences when interacting with an ontology. A user of an application ontology is normally not the creator of the ontology, and he or she may not even be knowledgeable about ontologies or ontology engineering. This observation was also true for the domain experts participating in the evaluation of the application ontology presented in this paper. Usability, as de ned and applied in this paper, is a necessary condition for an application ontology to be used on a regular basis. Users should not feel frustrated when attempting to understand the ontology. Instead, users should be able to put trust in the ontology and be con dent that they can carry out their tasks e ectively and e ciently while using the ontology.
The evaluation of the usability of a product or system has a long history and
back in 1986, Brooke developed a questionnaire, so called System Usability Scale
(SUS) [
In our project, the correctness of an ontology is probably the most important
quality feature that needs to be evaluated. Di erent de nitions of correctness
can be found in di erent articles, e.g. [
Competency questions (CQ) are among the available methods of gathering
information. They are natural language sentences that express patterns that
apply to a type of questions one would expect an ontology to be able to answer.
CQ have been suggested to serve as functional requirements of an ontology [
Ideally, correctness veri cation is an activity realised by a domain expert who not only grasps the details of the application itself but also has su cient knowledge of ontologies to be able to perform the evaluation. In many situations, this kind of human evaluator, with knowledge of ontologies, is not available. Hence, we suggest that a method, which is based on the verbalization of an ontology, provides support to the correctness evaluation process. In this process, the ontology is rst verbalized into a natural language text. Thereafter, the text can be read and understood by non-ontology experts, i.e. application domain experts, and compared with the information contained in the source information that was used to construct the ontology.
Applicability of an application ontology, in the context of this paper, is de ned
as the quality of the ontology regarding its appropriateness for a particular
application, task or purpose. The applicability concept includes the appropriateness of
the structure or semantics (e.g. the depth of the class hierarchy and the number
of instances), the appropriateness of the ontology language (i.e. which language
constructs that can be used in an ontology constructed with that language), and
the appropriateness of the ontology document (i.e. a particular serialization of an
ontology, such as RDF/XML [
In this section, we demonstrate the speci c details of the evaluation with regard to the three quality features, applying them on the software requirements ontology described in Section 2. We also describe how the methods and tools were applied during the evaluation, and discuss the evaluation results. Four industry domain experts (DE1-DE4) from the avionics domain (and speci cally from the Statements to evaluate the previous knowledge of ontologies, Protege and SRS 1 To what extent do you know about ontologies,
what they represent, how they can be used, etc.? 2 To what extent do you know about the ontology
tool, Protege, and how to use it? 3 To what extent are you familiar with SRS? 4 To what extent do have you been directly in
volved in developing di erent SRSs? 5 How familiar are you with the content of the
SRS used in this project? 6 To what extent have you been involved in the development of the software component de ned in the SRS? 2 1 software domain), and one ontology expert (OE) (not being the ontology developer), participated in the evaluation of the usability and the correctness of the application ontology. The applicability was evaluated by the ontology expert only, as he was the person who developed the ontology-based program for the test case generation.
To validate the evaluators' background competence in the eld of ontologies and software requirements speci cations (SRS), a quick questionnaire was applied. The results are presented in Table 1, where the grades range from 1, indicating "not at all", to 6, indicating "to a great extent". As can be observed in the table, none of the industry experts had a deeper knowledge of working with ontologies, apart from an overall understanding of the speci c application ontology as a result of the many presentations provided during the recurrent project meetings. Additionally, they did not have any previous experience of using ontology editors, such as Protege. However, these de ciencies were made up for by their experiences with SRS, both as designers and users. As a comparison, the ontology expert has a deep knowledge of ontologies, ontology tools and also the content of the SRS used in the project. This di erence in knowledge and experience is the reason why the results presented in the following two subsections look like they do. In spite of these di erences, by applying an appropriate preparation before the evaluation of an application ontology using adequate tools, such as Protege and verbalization, the negative e ects of such di erences can be reduced to a minimum.
In the evaluation of the usability of the application ontology, we applied a version
of the SUS that was introduced by Casellas in [
A sample of only ve evaluators could by some be considered as being too
small. Tullis and Stetson [
Since it was di cult for the industry experts to comprehend the ontology only by reading the raw ontology document (i.e. the ontology documented in formal ontology language), Protege was used as the tool to visualize the application ontology in the evaluation. Before the evaluation, a 40-minute tutorial on the application ontology and the Protege tool was provied to the 4 industry experts. During the evaluation itself, the questions in Table 2 were answered by all of the 5 experts. After having answered the questions, the score for each question was calculated (where the score can range from 0 to 4). For items 1, 3, 5, 7, and 9 (the positive statements) the score contribution was calculated as the scale position value minus 1. For items 2, 4, 6, 8, and 10 (the negative statements), the score contribution was calculated as 5 minus the scale position value. After having done
Evaluator # Requirements Evaluated Evaluation Time (in minutes)
SUS Scores
Grades this simple exercise for the 10 statements, the sum of the scores was multiplied with 2.5 to obtain the overall SUS scores.
The nal SUS scores are presented in Table 3 together with the grades,
accordingly to Bangor et al. [
To sum up, there exist strong indications that the usability of an ontology is not exclusively determined by the ontology complexity, but also by the tool used to read and comprehend the ontology. Before initiating the evaluation of the usability of an ontology, it is therefore recommended to commence with a thorough introduction of the tool that is going to be used. A good ontology visualisation tool, like VOWL, would also be of great help in this endeavour.
In this real application the information to be represented in the ontology is the
information written in the SRS document. Furthermore, the information in the
SRS document has been reviewed and validated. Therefore, the correctness is
evaluated as a whole, by comparing the information asserted in the ontology to
the information in SRS document. For this purpose, two di erent tools were used
by the ve evaluators. The rst tool that was used once again was Protege. The
evaluators used Protege to access the information represented in the ontology and
compared this information with the information written in the SRS document.
The second tool that was used was a web-based application, developed within
the project, that transformed the information represented in the ontology into a
natural language text in English. The web-based application is an ontology
verbalization tool with the goal of making ontologies more readable to non-ontology
experts [
The evaluation was organized in two parts. In the rst part, the ontology was evaluated relying solely on Protege. In the second part, the ontology was evaluated using only the verbalization tool. However, while using the verbalization tool, the evaluators could make use of Protege for further validation of details, if required. The evaluation using the verbalization tool proceeded as follows. The evaluators uploaded the ontology (in OWL format) to the tool, retrieved the verbalization of the ontology (in plain English), and compared the verbalization result with the written information in the SRS document. Figure 2 presents the verbalization of a piece of the application ontology (the left part of the gure) and the corresponding text found in the requirements document (the right part of the gure). The green colored texts in the left part of the gure indicate the prede ned patterns found in the verbalization algorithm. The remainder of the text (the black text found in the left part of the gure) comes from the ontology # Req.
Evaluated Eval. Time (in minutes)
Tool
Protege verbalization
Protege verbalization itself. The fragment of the ontology for requirement SRSRS4YY-219 is similar to requirement SRSRS4YY-431 presented in Figure 1.
In both parts of the evaluation, i.e. using Protege or verbalization, the domain experts were asked to validate the correctness of the ontology as compared to the text found in the SRS document. Each of the four industry domain experts were assigned a xed number of requirements that they were asked to validate (9 or 10 requirements each). Table 4 shows the number of requirements that were veri ed by each of the evaluators and the evaluation time needed while using either Protege or the verbalization tool. As can be observed in the table, in general the evaluators used less time but veri ed more requirements when using the verbalization tool. The reason for this is simple: the di culty when using Protege for the rst time, together with the staggering amount of information represented in the application ontology, causes a reduced evaluation speed. Some of these observed di erences could most likely be reduced by increasing the time spent on training the users in using Protege. As a secondary result, the evaluations sometimes indicated errors in the application ontology, errors that was subsequently removed by the ontology developers.
So, when should a domain expert rely on Protege for the evaluation of an ontology and when it is enough to rely on the verbalization of an ontology? Our conclusion is that, in most situations, it is su cient to verbalize an ontology to be able to detect possible errors or mismatches. And, as shown in Table 4, if time is of the essence, verbalizing an ontology normally beats relying on Protege, especially if the domain experts are not familiar with Protege. However, in some situations it is recommended to use both tools in a complementary fashion. Or, as one of the domain experts put it, "Note that 'human understandable' issues that are 'machine impossible' will go undetected [if relying solely on verbalization, authors comments]. In Protege it was possible to see when a requirement was misinterpreted. Here it is back to textual representations [when using verbalization, authors comments] that may hide the misunderstanding. Easier to read though, but changing format is sometimes good".
According to our de nition of applicability, the ontology should exhibit the quality of appropriateness when used for a particular application domain or purpose. The applicability of the ontology is evaluated by looking at its applicability in the task of automatically generating software test cases based on SRS.
In the case study, consisting of the communication software component,
software test cases were generated by a knowledge-based system consisting of a
knowledge base (i.e. the application ontology), a set of inference rules and an inference
engine. The knowledge base had to provide the following: 1) Domain knowledge
at the instance level su cient for the construction of test cases; 2) Domain
knowledge at the schema level necessary for domain-speci c reasoning; 3) Syntax and
semantics allowing for querying the knowledge base during the reasoning; and 4)
Documentation that allows a developer to understand the represented knowledge.
The inference rules used to create the test cases represent strategies for test case
generation that were acquired through an analysis of the case study and
discussions with the domain experts. The inference rules were implemented in Prolog
because it provided a built-in inference engine, which was used for reasoning. The
implementation details can be found in [
To be used as part of the knowledge-based system, the serialization of the
requirements ontology was suggested to follow the OWL functional-style syntax [
The conditions and actions of the inference rules included patterns that were matched against the statements in the knowledge base. The pattern matching was performed by Prolog, resulting in retrieval of instances from the application ontology, to check conditions of the rules as well as to construct di erent parts of test cases. The continued experiments showed that the instances contained in the application ontology were su cient for the execution of the inference rules and the generation of software test cases. During the rst experiment, 40 inference rules were used to generate 18 test cases. 66 distinct entities from the ontology were used for the test case construction. In the rst attempt, the test cases were generated as plain text in English. The experiment showed an almost one-to-one correspondence between the texts in the generated test cases and the texts provided by one of our industrial partners in the form of a Software Test Description (STD) document. It should be stressed, however, that the test cases could as easily have been generated as executable scripts, if so desired.
The evaluation showed that the developed application ontology ful lled its purpose. The ontology has been used for the automation of a part of the testing process and allowed for the successful generation of test cases. The ontology documentation contained an overview of the represented knowledge, which helped to bootstrap the development of the knowledge-based system. The syntax and semantics provided by the ontology satisfactorily supported the execution of the inference rules by the inference engine. The domain knowledge represented in the application ontology provided means for domain-speci c reasoning and allowed for the construction of test cases that corresponded to the sample test cases provided in the case study. Minor de ciencies in the application ontology, discovered during the development of the inference rules, were addressed and removed in the following iterations of the ontology development process. One example of a de ciency that was identi ed during the applicability evaluation, was the lack of distinction between single-value requirement parameters and enumeration parameters. This distinction is important for test case construction during the domainspeci c reasoning. The lack of explicit representation of the enumeration type of parameters led to the loss of domain semantics, something that had to be remedied by an extra inference rule checking the type of a requirement parameter. Because this knowledge is declarative rather than procedural, this distinction was introduced in the ontology in the next iteration. The result of this can be observed in Figure 1 as a "list-type" data value (the upper-left green box, "frs485Com, rs422Comg"). Another example of a modi cation of the application ontology, as a direct result of the applicability evaluation, was the division of the FIFO class into subclasses of transmission queues and reception queues.
Within the vast research eld that encompasses all aspects of ontology development is situated the crucial activity of building an ontology of "high quality". The challenge is the di culty in de ning quality, and decide on the methods/tools for the evaluation of quality. Currently, there exist no common de nition of quality and well-established evaluation methods/tools. In the work presented in this paper, we have evaluated three quality features, usability, correctness, and applicability, features that we consider critical for the evaluation of an application ontology, which includes knowledge from software requirements in support of the creation of software test cases. We have also presented the methods and tools requested for the evaluation of the three quality features, along with the results from these evaluations.
Correctness is probably the most important quality feature, and also the most di cult to evaluate. Characteristics, such as completeness, conciseness and consistency, can be considered as sub-features of correctness. In the real application study that has been presented in this paper, correctness of an ontology is evaluated as a whole. Correctness of the ontology is measured by comparing the information represented in the ontology to the information found in a requirements document. The evaluation was performed by four industry domain experts and one ontology expert. The results from the evaluation performed by the industry domain experts indicate that the use of a verbalization tool can drastically improve the productivity during the evaluation of an ontology. Nonetheless, the use of a tool like Protege is still needed to verify details in an application ontology. As the evaluations in this paper have indicated, extra time needs to be invested in the training of domain experts, for them to make full use of Protege.
Evaluation can be related to di erent attributes of an ontology, such as its structure and semantics, ontology language and ontology documentation. These attributes of an application ontology are very relevant to the task or application the ontology supports. As such, they can be covered in the evaluation of the applicability. It is likely that the requirements on the applicability might not be well-de ned when developing an ontology. Hence, the evaluation of the applicability has to be coordinated with the development of the application that uses the ontology. But, at the same time, the evaluation of the applicability helps to make ne-grained improvements of the ontology. These improvements make the ontology better adjusted to the application.
The usability of an application ontology is of utmost importance from an application experts' point of view. If one really wants the application experts to fully embrace ontologies and use them, the ontologies must be "user-friendly". In this paper we have shown that the System Usability Scale is useful when evaluating the usability of an ontology.
To provide good tools for the ontology quality evaluation is paramount for the e ectiveness and e ciency of ontology development. In the long term, the focus of our future work will be on investigating more examples of practical methods and tools for ontology quality evaluation. For the near future, we foresee a generalization of the verbalization process, such that the tool can be used for evaluating other types of ontologies.
The work presented in this paper was nanced by the Knowledge Foundation in Sweden, grant KKS-20140170.