Monitoring ontology quality is a key activity during the whole ontology lifecycle that requires adequate tools to provide overview over changing ontologies. In this paper we present an interactive dashboard framework for monitoring ontology metrics and quality indicators. While the framework is designed as generic, we present a prominent use-case for OBO Foundry ontologies, backing by ROBOT metrics and quality indicators.
Section 2 presents some existing solutions, and in section 3 we present the architecture of our solution together with its features. In section 4 we evaluate our work on a set of OBO Foundry ontologies and discusses its usability with some members of the OBO Foundry community and section 5 presents our conclusions and lessons learnt.
A traditional view on ontology quality analysis is given in [
Ontology quality dashboard could be configured also using general-purpose BI solutions over
RDF, like SANSA[
The Interactive Dashboard Framework is designed to deliver dynamic dashboards over a set of
ontologies that can change over time. Its architecture is depicted in Figure 1. One of the key
components is ROBOT [
For the frontend, the existing dashboard solution Kibana is utilized. Kibana [
One of the main features is the approach to obtaining quality data on ontologies. Since the
ROBOT tool is a key component, its capabilities were analyzed in detail, the most suitable
commands were: robot measure to generate metrics and robot report to get a report on
violations in ontologies. It was important for us to represent the outputs of these commands
in RDF format in order to save them in triple-store. In order to implement this, the first idea
was to describe these two commands using a suitable vocabulary. The results of the research of
suitable vocabulary were the finding of DQV(Data Quality Vocabulary)[
To standardize validation and generation of violation reports and keep them independent of
the ROBOT tool we use SHACL (Shapes Constraint Language) [
Here is an example of translating the ROBOT rule called "lowercase definition": PREFIX obo : < h t t p : / / p u r l . o b o l i b r a r y . o r g / obo / > SELECT DISTINCT ? e n t i t y ? p r o p e r t y ? v a l u e WHERE {
VALUES ? p r o p e r t y { obo : I A O _ 0 0 0 0 1 1 5 obo : I A O _ 0 0 0 0 6 0 0 } ? e n t i t y ? p r o p e r t y ? v a l u e .
FILTER ( ! r e g e x ( ? v a l u e , " ^ [ A−Z0 − 9 ] " ) )
FILTER ( ! i s B l a n k ( ? e n t i t y ) ) } ORDER BY ? e n t i t y
This rule can be represented in SHACL as follows: ex : l o w e r c a s e _ d e f i n i t i o n a sh : NodeShape ; sh : t a r g e t C l a s s owl : O b j e c t P r o p e r t y , owl : A n n o t a t i o n P r o p e r t y , owl : C l a s s ; sh : p r o p e r t y [ sh : p a t h obo : I A O _ 0 0 0 0 1 1 5 ; sh : s e v e r i t y sh : I n f o ; sh : m e s s a g e " l o w e r c a s e _ d e f i n i t i o n " ; sh : p a t t e r n " ^ [ A−Z0 − 9 ] ( . ∗ ) " ; ] ; sh : p r o p e r t y [ sh : p a t h obo : I A O _ 0 0 0 0 6 0 0 ; sh : s e v e r i t y sh : I n f o ; sh : m e s s a g e " l o w e r c a s e _ d e f i n i t i o n " ; sh : p a t t e r n " ^ [ A−Z0 − 9 ] ( . ∗ ) " ;
Not all the rules were described exactly as above, complexity of some rules made us replace sh:property with sh:sparql and adopt the original SPARQL query to its SHACL version. Nevertheless, as many SHACL rules as possible were left without using SPARQL in order to improve validation eficiency. Thus all the predefined queries of the ROBOT report command were transformed in this way, with the exception of two: "deprecated class reference" and "deprecated property reference" due to their complexity.
The Interactive Dashboard Framework solves the issue of ontology version tracking, which is currently a significant concern. Many ontology creators either overlook the using of specific OWL attributes like owl:version and owl:versionInfo, or they they use diferent schemata as numeric identifiers or date stamps, or a combination of thereof. This inconsistency can lead to dificulties in efectively tracking.
To solve this problem, the Interactive Dashboard Framework works in update mode. With each update, the framework performs a check to determine if the ontology contains date stamp attribute. If such an attribute is absent, the framework assigns its own version to the ontology data, which is date of update. This approach eliminates the reliance on inconsistent versioning practices and ensures ability to track changes over time.
By adopting this strategy, the framework enables the monitoring of variations in the number of violations or specific violations themselves across diferent ontology versions.
In order to test our framework, it was decided to make a prominent use case for OBO Foundry ontologies. The dashboard (available at http://tinyurl.com/obodashboard) currently contains a subset of all OBO Foundry ontologies, mainly to keep the experiment limited and running on a common hardware. While OBO Foundry provides its own dashboard solutions based on its principles, these solutions ofer more general information without delving into specifics, such as detailed data on the types of violations. Our dashboard for OBO Foundry ontologies ofers easy configuration and possibility to monitor ontology evolution in time. It consists of three main sections: "All ontologies", "Single ontology" and "Specific ontologies". In Figure 3 you can see part of "Single ontology" section, which demonstrates the possibility of selecting an ontology and its version, as well as information about violations of the ontology.
Thus, using the example of a dashboard for OBO Foundry ontologies, you can easily configure a dashboard, for this you need to perform only three steps: 1. Configure a SPARQL query tailored to your specific requirements. This query will retrieve the necessary data from GraphDB. By carefully designing your query, you can extract relevant data you wish to visualize. 2. Index data from GraphDB to Elasticsearch with RDF Indexer using the SPARQL query from the previous point. 3. Use Kibana to design customized visualizations, such as graphs, charts, and tables, to efectively represent the ontology data indexed in Elasticsearch.
We received feedback from the OBO Community by conducting four test scenarios that test usefulness where subjects gave comments and rated the scenario on a scale. And also ten questions on usability. Three subjects took part in the tests, and another subject gave his feedback with the format of the discussion.
The test results revealed that most of the comments were focused on the dashboard’s UI rather than the data it contains. The primary drawback highlighted by all participants was the height of the dashboard box, which required constant scrolling down to access the most relevant information. This was caused by the navigation bar taking up a large percentage of the available area. Additionally, Kibana has its own UI elements like data filtering fields that further reduce the available space for the dashboard content. The separation of sections in the navigation panel also raised doubts, most participants would like to see the "All ontologies" and "Specific ontologies" sections in one. Almost all participants had problems with filtering data from the first time, this is due to the fact that filtering in Kibana is quite specific and takes a little time to sort it out, as with all other things, since Kibana is a bit technical UI.
As for the positive qualities of the dashboard, the main feature that the community appreciated was the version of ontologies with the possibility of tracking the number of violations in chronological order. One participant was surprised at how we approached the method of generating a violation report, i.e. by transforming ROBOT rules into SHACL syntax, as well as the description of metrics generated by ROBOT using DQV. Another thing appreciated by the participants were the links to the ROBOT rules website and to the ontology repositories.
The presented solution shows in a flexible and configurable way to configure an interactive dashboard over ontologies stored in a triple-store (or external files), while still providing useful outputs to the domain experts, as our experiment showed.
In the future, we would like to address several research directions. First, making indexing of large ontologies more eficient and robust. Also, incorporating other types of ontology metrics and quality control rules than those provided by ROBOT. Last, but not least, creating a set of predefined quality control widgets for a default dashboard ready to provide basic quality-related information about any OWL ontology.
This plan for future development will also include integration with the OBO Dashboard, further enhancing its utility and accessibility, as well as trying to apply the dashboard solution to other ontology communities.