=Paper=
{{Paper
|id=Vol-2744/paper25
|storemode=property
|title=Non-Empirical Metrics for Ontology Visualizations Evaluation and Comparing
|pdfUrl=https://ceur-ws.org/Vol-2744/paper25.pdf
|volume=Vol-2744
|authors=Ildar Baimuratov,Than Nguyen
}}
==Non-Empirical Metrics for Ontology Visualizations Evaluation and Comparing==
https://ceur-ws.org/Vol-2744/paper25.pdf
Non-Empirical Metrics for Ontology Visualizations
Evaluation and Comparing?
Ildar Baimuratov[0000−0002−6573−131X] and Than Nguyen[0000−0002−6679−7839]
ITMO University, Kronverksky Pr. 49, bldg. A, St. Petersburg, 197101, Russia
{baimuratov.i,nguyenngocthan92}@gmail.com
Abstract. There are numerous ontology visualization systems, however, the choice
of a visualization system is non-trivial, as there is no method for evaluation and
comparing them, except for empirical experiments, that are subjective and costly.
In this research, we aim to develop non-empirical metrics for ontology visual-
izations evaluation and comparing. First, we propose several half-formal met-
rics that require expert evaluation. These metrics are completeness, semanticity,
and conservativeness. We apply the proposed metrics to evaluate and compare
VOWL and Logic Graphs visualization systems. And second, we develop a com-
pletely computable measure for the complexity of ontology visualizations, based
on graph theory and information theory. In particular, ontology visualizations are
considered as hypergraphs and the information measure is derived from the Hart-
ley function. The usage of the proposed information measure is exemplified by
the evaluation of visualizations of the sample of axioms from the DoCO ontology
in Logic Graphs and Graphol. These results can be practically applied for choos-
ing ontology visualization systems in general and regarding a particular ontology.
Keywords: Ontology Visualization, Expert Evaluation, Hypergraphs, Informa-
tion Measure.
1 Introduction
Visualization of an ontology improves comprehension of knowledge it contains. There
are numerous ontology visualization systems, the reviews are presented in [1–3]. How-
ever, the choice of a visualization system is non-trivial, as there is no method for evalu-
ation and comparing of ontology visualization systems present at the literature, except
for empirical experiments, that are subjective and costly.
Little researches consider the evaluation of visualization systems in general. In [4]
the authors propose some recommendations considering graph diagrams, like mini-
mization of crossings between edges. The authors of [5] present empirical research on
applying these criteria to automatic graph layout algorithms. In [6] several new shape-
based metrics are proposed for large graphs. All these metrics are based on empirical
experiments, i.e. on human assessments.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons Li-
cense Attribution 4.0 International (CC BY 4.0).
?
Publication is supported by RFBR grant 20-01-00358
2 I. Baimuratov and Than N.
Therefore, we aim to develop formal metrics for ontology visualizations estimation.
’Formal’ means that they must be objective and computable. Some metrics we propose
require external knowledge of the language being visualized, its semantics, and knowl-
edge of other visualization systems, therefore, they are half-formal and require expert
evaluation. Another criterion is based on graph theory and information theory and is
fully computable.
The outline of the paper is as follows: in Section 2 we propose several metrics
for expert evaluation and in Section 3 we derive the information measure for ontology
visualizations complexity.
2 Expert evaluation
First, we propose to consider several features of visualization systems that, though re-
lated to the formal properties, like completeness, still require expert evaluation, as they
involve external knowledge.
2.1 Completeness
The most important property of a visualization system is its completeness with respect
to the language being visualized, because if a visualization system can not represent
some axioms of an ontology, the system can not be applied to the ontology. In addition,
a common reference language serves as a common denominator for comparing different
visualization systems.
Ontologies are denoted on the OWL language [7]. The OWL 1 standard provided
three increasingly expressive sub-languages: OWL Lite, OWL DL, and OWL Full. In
this paper, we consider OWL DL language, as it provides the maximum expressiveness,
retaining decidability.
The formal foundation of OWL is description logics (DLs) [8]. DLs are a family
of logic languages, that can be used to represent the terminological knowledge of an
application domain. We consider axioms, formulated with the SHOIN description logic
syntax, as it corresponds to OWL DL language. We evaluate completeness of a visual-
ization system by counting the number of SHOIN syntax entities that the system can
represent.
2.2 Semanticity
We suppose, the advantage of a visualization with respect to a reference language is
that it improves comprehension of a formula with representing its semantic. Therefore,
we propose to evaluate the ability of a visualization system to represent semantics of
expressions.
We consider a diagram of a visualization system for a logical relation as semantical,
if it represents the semantic of the relation. For example, compare the visualization of
conjunction from Graphol [9], Fig. 1, with the corresponding Venn diagram [10], Fig. 2.
Venn diagram represents that these two sets have common elements, while in Graphol
conjunction is just labeled with a hexagon.
Non-Empirical Metrics for Ontology Visualizations Evaluation... 3
Fig. 1. Conjunction in Graphol Fig. 2. Conjunction in Venn diagrams
2.3 Conservativeness
Finally, we suppose that it is important to use existing graphic primitives from mathe-
matical theories, as in the other case, i.e. introducing new graphic primitives, instead of
helping a user to understand an ontology it forces him or her to learn just one more lan-
guage. Considering again the example above, in Graphol a user has to learn that hexagon
denotes conjunction, while if the Venn diagram was used, the user familiar with Venn
diagrams would have understood the diagram without additional instructions. There-
fore, we consider a visualization system as conservative, if it uses the existing graphic
primitives.
2.4 Example of evaluation
We provide an evaluation of the VOWL [11] visualization system as an example. We
examined its completeness with respect to OWL DL language, its semanticity, the abil-
ity to represent the semantics of relations, and whether its graphic primitives are new or
adopted from common visualization systems. See Table 1.
Here we see that VOWL can represent only 12 of 15 entities of the SHOIN descrip-
tion logic, therefore, its completeness rate is 0.8. The diagrams for concepts, conjunc-
tion, disjunction, and equivalence are semantical and conservative, as they are based on
Venn diagrams. The diagram for roles can be considered as graph-theory based, there-
fore, it is also semantical and conservative. The diagram for negation is conservative, as
it uses the sign of negation from logic, but it is not semantical since it doesn’t represent
the semantic of negation.
In addition, we provide an example of comparing of visualization systems, per-
formed by the authors. We compare VOWL with Logic graphs (LGs) [12], the seman-
tically oriented ontology visualization method, developed by us. The analogous evalu-
ation of the LGs is in Table 2, the scores of VOWL and LGs are presented in Table 3.
As wee see, LGs are complete and more semantical and conservative, than VOWL.
3 Information measuring
In the previous section, we proposed several metrics for expert-based evaluation. Prop-
erties like completeness and conservativeness are important for visualization systems
evaluation, but it is hard to imagine that they would be fully computable. Thus, we pro-
pose one more approach to ontology visualization systems evaluation intended to be
completely formal. This approach is based on information measuring.
4 I. Baimuratov and Than N.
Table 1. Expert-based evaluation of VOWL
N Entity Complete Semantical Conservative
1 Concept 1 1
2 Role 1 1
3 Negation 0 1
4 Conjunction 1 1
5 Disjunction 1 1
Existential
6 0 0
restriction
Universal
7 0 0
restriction
8 Transitive role 0 0
9 Inverse role 0 0
10 Role hierarchy 0 0
Number
11 0 0
restriction
12 Nominal 0 0
Functional
13 0 0
role
Concept
14 0 0
inclusion
Concept
15 1 1
equivalence
0.8 0.33 0.4
The intended information measure should estimate not the content of the ontology,
as it is the same for each visualization, but the complexity of its form. It implies the
following requirement for the intended information measure:
– as it should estimate the complexity of an ontology visualization, it should depend
on the complexity of its structure, in other words, on the number of nodes, edges,
and types of edges;
– it should be normalized, as visualizations of one and the same ontology in different
visualization systems can have a different number of nodes and edges;
– as it should measure the visualization complexity, a visualization with a greater
number of nodes, edges, or edges types should have a higher value of the measure.
Non-Empirical Metrics for Ontology Visualizations Evaluation... 5
Table 2. Expert-based evaluation of LGs
N Entity Complete Semantical Conservative
1 Concept 1 1
2 Role 1 1
3 Negation 1 1
4 Conjunction 1 1
5 Disjunction 1 1
Existential
6 1 1
restriction
Universal
7 1 1
restriction
8 Transitive role 1 1
9 Inverse role 1 1
10 Role hierarchy 1 1
Number
11 0 1
restriction
12 Nominal 1 1
Functional
13 0 1
role
Concept
14 0 0
inclusion
Concept
15 1 1
equivalence
1 0.8 0.93
3.1 Hypergraphs as the formal framework
Before defining the information measure, we have to define the formal framework. We
propose to consider ontology visualization as a hypergraph. Simple graphs are not suit-
able for our goals as many ontology visualization systems use edges connecting more
than two nodes. A hypergraph can be represented as an incidence matrix, therefore, an
ontology visualization can be represented as an incidence matrix as well.
6 I. Baimuratov and Than N.
Table 3. Expert-based comparing of VOWL and LG
Complete Semantical Conservative
VOWL 0.8 0.33 0.4
LGs 1 0.8 0.93
Let there is a hypergraph H = (X, E), where X – is a set of nodes and E is a set
of edges. It is represented with |X| × |E| incidence matrix A = (aij ), where
(
1, if xi ∈ ej
aij = (1)
0, otherwise
for undirected graph and
0
−1, if (xi , x ) ∈ ej
0
aij = 1, if (x , xi ) ∈ ej (2)
0, otherwise
and
aij = 2 if ej = (xi , xi ). (3)
Consider the following axiom from the Document Components Ontology (DoCO)
[13] as the example
chapter v ∃contains.paragraph t section (4)
and its visualization in the Graphol system, see Fig. 3. Here edges denoting disjunction
Fig. 3. Visualization of the axiom 1 in Graphol
connect three nodes: ’graphol.paragraph’, ’graphol.section’ and ’or’, therefore, it is the
hypergraph. The incidence matrix for this hypergraph is Table 4. Each node of the
diagram corresponds to a row of the matrix and each relation – to a column. As nodes
’graphol.paragraph’, ’graphol.section’ and ’or’ are connected with the edge ’or’, the
corresponding cells have value 1. In this research, we ignore the direction of edges for
simplicity.
Non-Empirical Metrics for Ontology Visualizations Evaluation... 7
Table 4. The incidence matrix for the axiom 1 in Graphol
subClassOf graphol.contains or
exists 1 1 0
graphol.chapter 1 0 0
graphol.contains 0 1 0
graphol.paragraph 0 0 1
graphol.section 0 0 1
or 0 1 1
3.2 The information measure
There are researches on graph information measuring. The survey of graph entropy
measures is in [14]. The authors of [15] perform information-theoretic analysis of edge
bundling visualizations in terms of adjacency matrices and mutual information. But
none of the measures presented there satisfy our requirement. Therefore, we develop a
new graph information measure.
First, we define the set of all different values of an incidence matrix A:
Definition 1. For a given incidence matrix A the set of all values is {a}.
For undirected graph {a} = {0, 1}, for directed {a} = {−1, 0, 1}, for directed graph
with loops {a} = {−1, 0, 1, 2} and so on.
Now we define the set of all possible edges E:
Definition 2. For a hypergraph H with a given set of nodes X the set of all possible
edges E = {a}X .
We are ready to define the information measure for hypergraph complexity estima-
tion by deriving it from the Hartley function [16]
logb |A|, (5)
where A is an arbitrary set and b – an arbitrary number. We substitute the number of
edges |E| as |A| and the number of all possible edges |E| as b.
Definition 3. For a hypergraph H with a given set of nodes X, a given set of edges
E and a set of corresponding incidence matrix values {a}, the information I(H) is
following:
1
I(H) = log|E| |E| = log|{a}| |E| (6)
|X|
Consider several simple graphs for illustration, see Fig. 4, its information estimation
is at the Table 5. As wee see, H2 has a more complex structure comparing to H1 and,
therefore, its information value is higher. H3 is directed and each directed edge con-
tains less information, therefore, with the same number of edges its information value
is lower compared to H1 . Summing up, the information measure satisfies the desired
properties.
8 I. Baimuratov and Than N.
H1 x2 H2 x2 H3 x2
e1 e1 e1
x1 x1 e3 x1
e2 e2 e2
x3 x3 x3
Fig. 4. The graph examples
Table 5. Information estimation of the graph examples
H |X| |E| |{a}| I(H)
1 3 2 2 0.33
2 3 3 2 0.53
3 3 3 3 0.21
3.3 Comparing visualizations with the information measure
We provide an example of comparing ontology visualizations with the developed in-
formation measure. Unlike expert-based evaluation, where we compared visualization
systems itself, for information measure we have to compare visualizations of a par-
ticular ontology. We use the DoCO ontology as it is a real ontology, used in different
applications, and it contains nontrivial axioms. We visualized some axioms of this on-
tology in Graphol and Logic Graphs (LGs) [12], the semantically oriented ontology
visualization method, developed by us. The list of axioms and their visualizations are
in Table 6.
The example of an incidence matrix for Graphol was provided in Table 4. Now
consider the example of the incidence matrix for Logic Graphs. The incidence matrix
for the axiom 1 in Logic graphs is Table 7.
We compare LGs with Graphol by measuring information of the corresponding vi-
sualizations for the sample of axioms, presented in Table 6. The result is in Table 8. As
we see, the average information of LGs on this sample is higher, than of Graphol.
4 Conclusion
In this research, we proposed several non-empirical metrics for ontology visualization
evaluation and comparing. These metrics are divided into two groups. The first group
includes three metrics: completeness, semanticity, and conservativeness. These metrics
require expert evaluation and, therefore, they are half-formal. As an example, we com-
pared two ontology visualization systems: VOWL and Logic Graphs.
The second group consists of the completely computable information measure, de-
rived from the Hartley formula, that allows normalized measuring complexity of ontol-
ogy visualizations, represented as hypergraphs with incidence matrices. As an example,
we compared Logic Graphs with Graphol by measuring average information of visual-
izations of the sample of axioms from the DoCO ontology.
Non-Empirical Metrics for Ontology Visualizations Evaluation... 9
Table 6. Visualizations of DoCO in LGs and Graphol
1 chapter v ∃contains.paragraph t section
2 abstract v (chapter t section) u (∃ispartof.bodymatter t f rontmatter)
3 af terword v section u ∃ispartof.backmatter)
4 appendix v (section u headedcontainer) u (∃ispartof.backmatter)
5 backmatter v discourseelement u container
6 chapterlabel v ¬sectionlabel
7 chaptersubtitle v ∃ispartof.chapter
8 f igure v mata t milestone
9 glossary v section u (∃ispartof.backmatter t f rontmatter)
Table 7. The incidence matrix for axiom 1 in LGs
subClassOf contains negation 1 negation 2 negation 3
chapter 1 0 0 0 0
conjunction 0 1 0 0 1
domain 1 1 0 0 0
paragraph 0 0 1 0 0
section 0 0 0 1 0
10I. Baimuratov and Than N.
Table 8. Comparing information of ontology visualizations
N LG Graphol
|X| |E| |{a}| I(H) |X| |E| |{a}| I(H)
1 5 6 2 0.52 6 3 2 0.26
2 9 11 2 0.38 10 5 2 0.23
3 5 3 2 0.32 6 3 2 0.26
4 7 4 2 0.29 8 4 2 0.25
5 4 2 2 0.25 4 2 2 0.25
6 2 2 2 0.5 3 2 2 0.33
7 3 2 2 0.33 4 2 2 0.25
8 4 5 2 0.58 4 2 2 0.25
9 7 7 2 0.4 8 4 2 0.25
0.4 0.26
These results can be practically applied for choosing ontology visualization systems
in general and regarding a particular ontology. Considering the presented examples,
it is recommended to use LGs rather than VOWL in general, as it has higher scores
of completeness, semanticity, and conservativeness, and for visualizing the mentioned
fragment of the DoCO, as LGs has higher informativeness.
References
1. Antoniazzi F., Viola F.: RDF graph visualization tools: A survey //2018 23rd Conference of
Open Innovations Association (FRUCT).IEEE,25-36 (2018)
2. Dudáš, M.,Lohmann, S., Svátek, V., Pavlov, D.: Ontology visualization methods and tools:
A survey of the state of the art. The Knowledge Engineering Review, vol 33, (2018)
3. Steffen, L., Stefan, N. , Florian, H. Thomas, E: Visualizing ontologies with VOWL. Semantic
Web. 7. 399-419. (2016)
4. R. Tamassia, C. Batini, and M. Talamo. An algorithm for automatic layout of entity-
relationship diagrams. In C. G. Davis, S. Jajodia, P. A. Ng, and R. T. Yeh, editors, 3rd Int.
Conf. on Entity-Relationship Approach (ER’83), pages 421-439. North-Holland, 1983.
5. Purchase, Helen & Carrington, David & Allder, Jo-Anne. (2002). Empirical Evaluation of
Aesthetics-based Graph Layout. Empirical Software Engineering. 7.
6. Eades, Peter & Hong, Seok-Hee & Klein, Karsten & Nguyen, An. (2015). Shape-Based
Quality Metrics for Large Graph Visualization. 502-514.
7. Deborah, L., Frank, V.,, et al. Owl web ontology language overview. W3C recommendation,
10(10):2004,(2004)
8. Baader, F.,Calvanese, D. , McGuinness, D. ,Nardi, D. and Patel-Schneider,P.: The Descrip-
tion Logic Handbook: Theory, Implementation, and Applications. Cambridge University
Press, second edition, (2007)
9. Graphol official website, Http://www.obdasystems.com/graphol. Last accessed 4 jul 2020
10. Ruskey, F.,Weston, M.:A Survey of Venn Diagrams. The Electronic Journal of Combina-
torics, (2005)
11. VOWL official website, Http://vowl.visualdataweb.org/. Last accessed 4 jul 2020
12. Mouromtsev, D.,Baimuratov, I.: Logic Graphs: A complete visualization method for logical
languages based on Ch. S. Peirce’s existential graphs. CEUR Workshop Proceedings, Vol.
2344, 2019, pp. 1-10.
Non-Empirical Metrics for Ontology Visualizations Evaluation... 11
13. Ontology DoCO, https://sparontologies.github.io/doco/current/doco.html. Last accessed 4
jul 2020
14. Dehmer, Matthias & Mowshowitz, Abbe. (2011). A History of Graph Entropy Measures.
Information Sciences. 1. 57-78.
15. Wu, Jieting & Zhu, Feiyu & Liu, Xin & Yu, Hongfeng. (2018). An Information-Theoretic
Framework for Evaluating Edge Bundling Visualization. Entropy. 20. 625.
16. R. V. L. Hartley, ”Transmission of information,” in The Bell System Technical Journal, vol.
7, no. 3, pp. 535-563, July 1928.