-The Chemical Entities of Biological Interest (ChEBI) ontology is an important knowledge source of chemical entities in a biological context. ChEBI is large and complex, making it almost impossible to be error-free, given the scarce resources for quality assurance (QA). We present a methodology to locate concepts in ChEBI with a high probability of being erroneous. An Abstraction Network, which provides a compact summarization of an ontology, supports the methodology. By investigating a sample of ChEBI concepts, we show that uncommonly modeled concepts residing in small units of the Abstraction Network of ChEBI are statistically significantly more likely to have errors than other concepts. The finding may guide ChEBI ontology curators to focus their limited QA resources on such concepts to achieve a better QA yield. Furthermore, this study, combined with previous work, contributes to progress in showing that this methodology can be applied to a whole family of similar ontologies.
INTRODUCTION
The Chemical Entities of Biological Interest (ChEBI)
ontology [
Quality assurance (QA) is an essential part of the ontology
lifecycle to make sure that there is no modeling error in an
ontology [
One practical ontology QA approach is to identify sets of
concepts, which are more likely to have errors than other
concepts. Focusing audits on such sets of concepts can achieve
a high QA yield in terms of the ratio of the number of concepts
with modeling errors to the number of reviewed concepts. Thus,
the question is how can we find such sets of concepts?
The SABOC team has previously demonstrated that
Abstraction Networks are an effective tool to identify sets of
concepts in ontologies that are more likely to have errors [
A partial-area (explained in the Background section) represents a group of concepts with the same structure and semantics, that is, all have the same set of relationships and are descendants of the same concept. Concepts in partial-areas that summarize few concepts (“small partial-areas”) expose their uncommon modeling, which is of special interest to us.
A modeling error in an ontology can be either an omission
error or a commission error. Examples of omissions are missing
a parent concept or missing a lateral relationship. Examples of
commission errors are an incorrect parent or an incorrect lateral
relationship. Omission errors represent knowledge in an
ontology that is not complete, while commission errors mean
that the modeling is wrong. Some ontologies, e.g. NCIt, are
intentionally modeled with incomplete knowledge. Hence,
commission errors are more severe and attract more interest
from ontology curators. A previous study [
The purpose of this study is to test whether the partial-area taxonomy-based QA methodology, focusing on concepts in small partial-areas, improves the error yield for the large Chemical Entity hierarchy of ChEBI.
II.
ChEBI [
ChEBI is provided in the W3C standard Web Ontology
Language (OWL) and OBO formats. In this study, using the
OWL format, object properties (relationships) are used only in
restrictions. There is a stated version and an inferred version of
OWL files. ChEBI provides the stated version. The reasoner
HermiT [
The ChEBI ontology consists of three hierarchies. In this study we used concepts from the largest, the Chemical Entity hierarchy, (98.7% of ChEBI concepts). It annotates chemical compounds within molecular entities. Besides the IS-A relationships, ChEBI’s Chemical Entity hierarchy has nine lateral semantic relationship types, such as has part, has role, and is conjugate base of. The other two hierarchies are the Subatomic Particle and the Role hierarchy. The former hierarchy is mainly used to categorize particles smaller than atoms, while the Role hierarchy defines the roles of molecular entities in different contexts. The ChEBI ontology employs a star rating annotation: “3-star” indicates that a concept was manually annotated by the ChEBI curator team; “2-star” means that the concept is annotated by a third party; and “1-star” usually represents a concept marked as deleted or obsolete. Thus, “1-star” concepts are excluded from our research.
QA requests by users of ChEBI are easily made via ChEBI’s GitHub issue tracking system (https://github.com/ebichebi/ChEBI/issues). ChEBI’s curators review and verify these requests. Approved changes are made available in subsequent releases. For example, a user may report a wrong reference to a certain compound, or a wrong definition or relationship for a compound, by creating an issue report on GitHub. These reports are reviewed on a weekly or monthly basis. If the curators agree with a request, they make the corresponding changes to ChEBI. The volume of requests and the response time reflect the limited QA resources of ChEBI’s curator team. At the time of this writing, there are 289 accumulated open issues and 3175 closed requests.
In our previous research, Abstraction Networks (AbN) have
been proven successful for summarizing and visualizing
ontologies [
Fig. 1(a) depicts a subhierarchy of 17 concepts (drawn as ellipses, labeled with their names). The arrows denote IS-A links. Concepts with the same set of lateral relationships are grouped together in a dashed bubble, labeled by the common set of relationships. For example, Atom, Nonmetal atom, Sblock element atom, Polymer, Ionic Polymer, and Polyanionic polymer are grouped in the green bubble because they all have only the has part relationship.
An area taxonomy is an AbN which consists of nodes called
areas and child-of links connecting areas. An area, depicted as
a color-coded box, represents the group of concepts with the
same set of relationships, i.e. in the same bubble in Fig. 1(a).
Fig. 1(b) shows the area taxonomy for Fig. 1(a). An area is
labeled by its set of relationship types. The concepts in the green
bubble are represented compactly by the green area {has part}.
The concept Chemical Entity and its descendants in the grey
bubble are now represented by the area {Ø}. (The symbol Ø
represents the empty set.) Similarly, the concept Hydrogen
atom and its descendants are represented by the area {has part,
has role}. Areas with the same number of relationship types
have the same color and are aligned at the same level. For
example, the areas {has part, has role} and {has part, is
conjugate base of} appear in the third level, colored in blue.
Child-of links (arrows in Fig. 1(b)) are derived based on the
underlying IS-A relationships in ontologies. See [
Areas summarize concepts with the same structure
(relationships). An area may have multiple roots, which are
concepts such that their parents are not in the area. For example,
Atom and Polymer in the green area {has part} are roots, each
imposing its semantics on its descendants in the area. For
example, the descendants of Atom are kinds of atoms and the
descendants of Polymer are kinds of polymers. The partial-area
taxonomy is a refinement of the area taxonomy. A partial-area
is composed of an area root concept and all its descendant
concepts in the same area. The size of a partial-area is the
number of concepts in it. Each partial-area is labeled by the
name of its root concept, expressing its semantics, with its size
in parentheses. Partial-areas are shown as white boxes in Fig.
1(c). For example, Atom (3) is a partial-area in the green area
summarizing three concepts, Atom and its two children, in the
green area. A partial-area taxonomy is an AbN composed of
nodes called partial-areas and hierarchical child-of links
(arrows in Fig. 1(c)) connecting them. The compact
summarization and visual simplification provided by the
partial-area taxonomy, make it easier to identify anomalies in
modeling the ontology. An area taxonomy and a partial-area
taxonomy can be created automatically by a software tool called
Ontology Abstraction Framework (OAF) [
III.
METHODS
As described above, there are two kinds of modeling errors in an ontology: omission errors and commission errors. Table I shows one omission error example and five commission error examples. For example, 3-buten-1-amine in row 1 is missing the relationship is conjugate base of (an omission error). In row 4, (S)-3-hydroxybutyric acid has an incorrect hierarchical relationship (a commission error). In this study the term “error” will be reserved for commission errors, unless otherwise noted, following the preference of ontology curators described above.
Given the fact that concepts within the same partial-area share the same structure and semantics, a partial-area that accommodates just a few concepts stands out as an “outlier,” which often needs more QA attention. We consider concepts as “uncommonly modeled” if they appear as outliers through the lens of the partial-area taxonomy. The motivation for auditing concepts in outlier sets is that if a concept is in a small partialarea while related concepts reside in large partial-areas, this raises suspicions about the correctness of the modeling of the concepts in the outlier set. For example, 421 polymer concepts with the same modeling in one partial-area appear to be correctly modeled. However, a partial-area with only two concepts (out of thousands of concepts in the ontology) may indicate error(s). It is, of course, possible that these two uncommonly modeled concepts are correct, but there is a higher possibility that they have errors. Once any errors are corrected, these concepts may become part of another (larger) partial-area. We formulate the following hypothesis.
Hypothesis 1: There exists a threshold value Θ differentiating small and large partial-areas, such that concepts in small partial-areas within the partial-area taxonomy for an ontology have a statistically significantly higher error rate than concepts in large partial-areas.
If Hypothesis 1 is confirmed even for one threshold value, it can be the basis for a QA methodology to guide the ChEBI ontology curators to focus on concepts in the small partial-areas whenever the QA resources are limited. To test the above Hypothesis 1, 500 concepts (0.5%) were randomly selected from the Chemical Entity hierarchy of ChEBI’s July 2017 release. Concepts were presented in random order to our domain expert LC for review. LC is a chemistry professor with substantial experience in chemical ontology auditing. We analyzed the sizes of partial-areas of these 500 concepts. The evaluation of Hypothesis 1 depends on the threshold value differentiating small partial-areas from large ones.
The threshold Θ may be different for different ontologies.
We consider two ways to obtain a threshold value. One is that
the threshold is predefined, based on prior experience from
other ontologies. Then we can conduct a study to test whether
we achieve statistical significance for the error rate difference
between small and large partial-areas. For example, in the study
by Zheng et al. [
The other way is to choose the threshold value that maximizes the error rate difference between small partial-areas and large ones. Hence, the threshold is determined by the study results. In this study, we introduce this second method. Considering the variations in terms of the numbers of concepts for different partial-area sizes, we use the weighted average error rate instead of the average error rate to determine the desired threshold. We call such a threshold an optimizing threshold, since it optimizes the difference between the weighted error rates of the two ranges of partial-area sizes. The weighted average error rate is calculated using formula (1), where is the total number of concepts in the partial-areas with the size i. is the commission error rate of the reviewed concepts in partial-areas with the size = i. ̅ is the weighted average commission error rate of all reviewed concepts of the partial-areas with sizes ranging over all existing sizes of small (large) partial-areas, respectively. Thus, the contribution of the concepts in the partial-areas of size i, to the weighted average error rate is ∗ .
∑ ̅ = ∑ ∗ ()
∑
We calculated the weighted average error rate for all
possible threshold values and picked the maximizing threshold
as value for Θ. We calculated the two-tailed p-value of Fisher’s
exact test [
IV.
RESULT
The Chemical Entity hierarchy had 106,707 concepts in the
July 2017 release of ChEBI. There are 27,498 partial-areas in
its partial-area taxonomy, from which an excerpt of 164
partialareas summarizing 92,685 concepts (86.86%) is shown in Fig.
2 created by the OAF tool [
Table II shows the distribution of concepts and errors in terms of partial-area sizes in the partial-area taxonomy. For each partial-area size i, the columns include the number of concepts , the number of audited concepts, the number of concepts with commission errors, and the corresponding error rate . For example, there are 25,798 partial-areas of size = 1, in which 236 concepts were reviewed by LC of which 98 concepts (41.53%) were found to have commission errors. Similarly, there are nine concepts with commission errors out of all 20 audited concepts (45.00%) in partial-areas with size 2.
The last three columns of Table II show the weighted error rate for the small partial-areas, for the large partial-areas, and the error rate difference between the two categories according to the corresponding threshold value equal to the partial-area size in the corresponding row. For example, in row 2, the partial-area size 2 is selected as the threshold to distinguish small partial-areas and large partial-areas. The weighted error rate for the small partial-area category according to formula (1) is 41.71%, while the weighted error rate for the large partialarea category is 26.31%. The error rate difference between them is 15.41%. From Table II we can see that the maximizing threshold value is 2. Thus, we choose Θ = 2 for Hypothesis 1.
Table III is the 2x2 contingency table for the commission erroneous concepts of the small partial-areas and the large partial-areas where the threshold value is the maximizing threshold 2. The count for erroneous concepts and concepts without errors for partial-areas is calculated using data from Table II. According to Table II, 236 concepts in the sample are from partial-areas with size = 1 (row 1) and 20 concepts are from partial-areas with size = 2 (row 2), thus a total number of 256 (=236+20) concepts from small partial-areas were audited. There are 107 (=98+9) concepts with commission errors in the small partial-areas. There are 149 (=256-107) concepts without commission errors from the small partial-areas. Similarly, we can calculate the number of erroneous concepts and the number of concepts without errors for the large partial-areas with size > 2, i.e., 57 and 163.
The two-tailed p-value of Fisher’s exact test [
Two types of commission errors were reported by our expert: 71 concepts (14.92%) had incorrect hierarchical relationships and 93 (19.54%) had incorrect relationship targets. Table I shows examples with their error descriptions. For example, in row 2, the incorrect target of the relationship is conjugate acid of for N-acetyl-D-glucosaminyldiphosphodolichol is a concept with 2- charge. It should have a 1- charge, since only one proton is removed. In row 5, zorbamycin is a secondary, not a primary amide, thus it has an incorrect hierarchical relationship error. The distribution of ChEBI concepts and errors between “2-star” and “3-star” concepts is in Table IV. It shows a much higher error rate for “3-star” concepts. Thus, we recommend to start by auditing “3-star” concepts in small partial-areas.
We utilized the partial-area taxonomy of ChEBI’s Chemical Entity hierarchy to explore the QA methodology focusing on small partial-areas. The results show that a threshold value Θ = 2 maximizes the average error rate difference between small and large partial-areas. The weighted average error rate for concepts of small partial-areas of up to 2 concepts is 41.71%. Hence, if the total of 27,262 concepts of the small partial-areas would be reviewed, about 11,371 concepts are expected to require corrections. Thus, if the QA resources are too limited to review 27,262 concepts, then 41.71% of the concepts from small partial-areas reviewed are expected to be erroneous.
Ochs et al. [
In the previous study on NCIt’s Biological Process
hierarchy by Hua et al. [
The concerns of SNOMED CT users about errors are
documented in [
According to Ochs et al. [
CONCLUSION
Abstraction Networks of ontologies have been proven to define a framework for the identification of concept sets that are expected to have comparatively higher error rates. Small partial-areas in the partial-area taxonomy derived from an ontology likely reflect uncommonly modeled concepts in the ontology. In this paper we tested the QA methodology that concentrates on auditing the concepts in small partial-areas.
This study applied the small partial-area-based QA methodology to the ChEBI ontology. Our analysis revealed that small partial-areas have statistically significantly more errors than large partial-areas, with an optimal threshold of two. The results confirmed that in the ChEBI ontology small partial-areas with size up to two concepts harbor statistically significantly more commission errors compared to large partial-areas. Overall, this approach narrows down the places in the ChEBI ontology where limited QA efforts should be invested to obtain a higher QA yield. This study, in combination with three other previous studies, provides progress toward showing that the small partial-area-based methodology is successful in identifying likely errors for a whole family of 76 BioPortal ontologies.
Research reported in this publication was partially supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA190779. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.