Ontology evaluation can promote the quality of an ontology as well as contextualize and facilitate its use in specific situations. Evaluating an ontology is a complex process, and often involves manual and time-consuming efforts. To address the complexity of ontology evaluation, many ontology evaluation frameworks have been developed. These frameworks provide structured methodologies, criteria, and tools to assess the quality of an ontology across various dimensions, such as accuracy, completeness, consistency, and relevance to a particular domain. The evaluation process can be categorized into different approaches, including structural, functional, and usability evaluations. Structural evaluation focuses on the internal consistency and logical coherence of the ontology's structure, ensuring that relationships between concepts are accurately represented. Functional evaluation, on the other hand, examines how well the ontology performs in specific tasks or applications, such as information retrieval or data integration. Usability evaluation assesses the ontology's ease of use, accessibility, and documentation, which are crucial for end-users and developers [1,2].
Ontology validation is a critical aspect of the ontology evaluation process. Validation focuses on ensuring that an ontology faithfully represents the intended domain. It involves verifying that the ontology's structure, concepts, relationships, and rules are correct, coherent, and logically consistent. This step is essential for guaranteeing that the ontology can be reliably used in applications such as data integration, semantic search, and knowledge management. Some tools like reasoners (e.g., Pellet, HermiT) and ontology editors (e.g., Protégé) provide functionalities to validate ontologies. Most of the ontology validation work involves human expert review to ensure that the ontology correctly captures the domain knowledge. Experts can provide insights into whether the ontology's concepts and relationships are properly modeled and relevant to the domain. The common practice of a human expert to re-examine and validate ontology development typically involves visualizing the ontology's structure. This is done by creating diagrams that illustrate the classes and their relationships as defined in the ontology. These diagrams serve as a crucial tool for understanding and verifying how well the ontology models the designated domain. The expert then compares this diagrammatic representation to use case scenarios, ensuring that the ontology captures the necessary concepts and relationships relevant to the specific application. This manual process is timeconsuming, error-prone and has additional challenges with respect to scalability, and efficiency, particularly in iterative development of an ontology. Computer-aided conceptual modeling can provide additional discipline and rigor to the manual, validation process.
PROVE Tool is an open-source process modelling tool, driven by a metamodel [3]. It codifies a pertinent process design domain framework [4], which has previously been established as effective in underpinning the evaluation of process descriptions in research [5]. PROVE Tool allows generating metamodel-compliant conceptual models of processes. It is designed to assist in the design and analysis of process descriptions [6]. The tool provides an environment where users can create detailed process models. These models describe the activities and artifacts involved in various processes as well as their hierarchical arrangement and flow. One of the primary features of PROVE Tool is its ability to verify process models based on its metamodel of the process. The tool allows its users to check for logical consistency, ensuring that the processes are free from errors such as unreachable states, incorrect scope of process hierarchy or contradictions. It also allows to check that the process designs fulfil specified requirements to ensure they provide intended results, in the form of artifacts and related achievements.
In what follows, we describe our proof of concept using PROVE Tool to validate that the design of the Informed Consent Ontology (ICO) [7,8]. The significance of this work to the conceptual modeling field is the demonstration of how conceptual modeling -particularly domain specific conceptual modeling -can underpin rigorous validation of formal ontologies. Further automation of the approach can significantly contribute to ontology evaluation.
Informed consent is a fundamental ethical and legal requirement in many fields, especially in research involving human participants, clinical trials, and medical treatments. It ensures that individuals are fully aware of and understand the nature, purpose, risks, benefits, and alternatives of a procedure or study before agreeing to participate. The ICO has modeled pertinent informed consent processes, including: giving a subject adequate information of the study, providing adequate opportunity for the subject to consider all options, ensuring that the subject has comprehended this information, obtaining the subject's voluntary agreement to participate and to continuingly provide information as the subject or situation requires, and archiving the signed documents for possible future usage [7]. The ICO modeling follows the Basic Formal Ontology based approach. Particularly, the "planned process" defined in the Ontology for Biomedical Investigation is the top-level class for the entire informed consent framework.
Using PROVE Tool, processes of the Informed Consent Ontology -as shown in Figure 4 of [6]were re-modeled and illustrated as diagrams, as shown in Figure 1. Through this modeling process, the human experts were able to quickly identify some deficiencies in the ontological modeling (of ICO): 1) Informed consent form states are not explicitly defined. Specifically, this does not force "Informed consent form approval" to occur before "Informed consent process" (as the artifact required for the latter -"Informed consent form" -is already available as the output of another process -"Informed consent form design"); 2) "Signed informed consent form" is an artifact in a specific state. This can be improved by attributing a state to the previously designed and approved artifact, and the process scope can be communicated using this; 3) There are no expectations stated from the various activities within "Informed consent process," as indicated by the lack of an outgoing arrow the in lower diagram in Figure 1. Figure 2 shows the potential improvement of the informed consent processes, based on our PROVE Tool analysis. Different states of the informed consent form after each specific process can be explicitly identified, such as "designed informed consent form", "approved informed consent form" and "signed informed consent form" as the output of the "Informed consent form design," "Informed consent form approval" and "Informed consent process" processes respectively. Furthermore, PROVE Tool can also provide the interaction of the completed implementation of a specific process by marking an artifact's status as "achieved". Figure 3 shows the "designed" state of the informed consent form marked in green once the "status achieved" being clicked. This green highlighted status is automatically shown in various PROVE Tool representations (in the user interface).
Ontologies play a significant role in our understanding of domains as well as in depicting desired behavior of systems in those domains. The validation of ontologies remains a labor intensive, error prone task. Here, for the first time, we utilized PROVE Tool to validate an ontology by rigorous, metamodel-based conceptual modeling of its processes. This proof-of-concept, computer-aided validation of the Informed Consent Ontology (ICO) shows the benefit of PROVE Tool's multiple graphical representations and perspectives in designing and evaluating the ontology. Specifically, the tool aids the human experts to quickly validate the ontological model using multiple, coherent diagrams. It is therefore promising to further develop PROVE Tool's capabilities as a user-friendly ontology validation tool or vice versa, using PROVE Tool to assist ontology design, particularly in process-centered ontologies that can benefit from the tool's ability to effectively capture and analyze process models. Future work can further extend the tool to align the underlying metamodel with the Basic Formal Ontology based planned process modeling, and to evaluate other ontologies using the tool. Currently the modeling of an ontology using PROVE Tool is performed manually. We plan to automate this modeling, e.g., by automatic transformation of OWL or RDF files into PROVE models and vice versa, which will facilitate ontology validation and re-design.