-Ontologies are important components of many health-information systems. The Chemical Entities of Biological Interest (ChEBI) ontology has become a standard reference for chemicals appearing in biological contexts. As such, assuring the quality of its content is imperative. In fact, ChEBI has a dedicated Web page at which errors and inconsistencies in its concepts can be reported. A study of the correctness of a random sample of ChEBI concepts is carried out. The results show that quite a large number of ChEBI concepts suffer from some kind of problematic modeling. For example, we found that 15.5% of the sample concepts exhibited severe errors of commission, including incorrect hierarchical (is a) and lateral relationships. Errors of omission were also prevalent. The overall results of our quality-assurance (QA) study are presented. Suggestions for enhancing the QA processes in place for ChEBI are discussed.
Ontologies are structures that capture terminological
knowledge for some target domain. Typically large in size and
high in complexity, ontologies have become fundamental
fixtures of health and biological information processing
environments. The Chemical Entities of Biological Interest
(ChEBI) ontology [
Due to their scope and complexity, it is nearly impossible
for ontologies such as ChEBI to be free of modeling errors and
inconsistencies. This can hinder their usefulness and adversely
affect the software systems and applications dependent on
them. ChEBI has been used, for example, as a source for
annotations in various bioinformatics databases, including
UniProt which utilized ChEBI for the cofactor comments
related to enzymes [
In this paper, we are interested in assessing the percentage of ChEBI’s concepts that suffer from some type of modeling issues. We refer to concepts that have errors or inconsistencies in their modeling as being erroneous concepts. We selected a random sample of 400 concepts for our study. Two subjectdomain experts in the field of chemistry were asked to carry out separate quality-assurance (QA) analyses of the entire sample and then produce a consensus report. The findings revealed that a substantial number of ChEBI concepts suffer from errors of commission and omission, suggesting the need for a formal initiative to map out an ongoing QA project for improvement of the quality of the modeling in ChEBI. Further recommendations for such a project are discussed.
A QA study of a random sample of ChEBI concepts was
carried out. ChEBI is updated monthly, and the version we
used in this study was that of February 2016, which contained
61,895 concepts, including 47,752 fully annotated compounds.
The sample was chosen using the basic sampling technique,
simple random sampling without replacement [
It should be noted that ChEBI employs three hierarchies to classify molecular entities. Its chemical entity hierarchy, the largest with 60,537 concepts (97.8% of all ChEBI concepts), is used to classify molecular entities according to their chemical structure. The subatomic particle hierarchy with 42 concepts categorizes particles smaller than atoms. ChEBI’s third hierarchy, the role hierarchy with 1,322 concepts, itself has three subhierarchies that define the roles in different contexts for the compounds, namely, the application subhierarchy to represent the intended use by humans for the compounds (e.g., fuel and anti-inflammatory agent), the biological role subhierarchy to represent the roles of compounds within the biological context (e.g., growth regulator and inhibitor), and the chemical role subhierarchy (e.g., acid and base). Note that there are six concepts belonging to both the chemical entity and subatomic particle hierarchies, e.g., helion and proton. The concepts randomly selected for our study came from all three of the main hierarchies, irrespective of hierarchy.
The QA analysis was done by a pair of chemistry subjectdomain experts. In the initial step, each concept from the sample was inspected by each of the experts separately— without any communication between them. Their results were tabulated in two individual error reports. Within a report, the rationale for the judgment of any error was recorded, and a suggested correction was proffered. Afterward, a combined report was prepared listing the respective findings of both experts for all the concepts. This report was shared with both experts who were then each asked separately to mark their agreement or disagreement with the findings of the other person—and to review their own findings in light of the other’s. After a review of the other’s report, each expert was able to change their mind regarding their own judgment of a modeling error. A concept previously judged to be modeled in error could instead be deemed to be correct, and vice versa. After this step of consensus building, a concept was deemed to be erroneous if both subject-domain experts agree that it was such.
Most of the QA analysis of ChEBI centered on issues with
concepts’ relationships. The three primary relationships in
ChEBI are the hierarchical is a relationship, capturing
standard subsumption in hierarchies, the relationships has
part, indicating the whole/part association between
compounds, and has role, linking concepts in the chemical
entity hierarchy to concepts in the role hierarchy. There are
seven chemistry-specific lateral relationships, namely, is
conjugate base of, is conjugate acid of, is tautomer of, is
enantiomer of, has functional parent, has parent hydride, and
is substituent group from. In combination, the relationships in
ChEBI can form converses. For example, if concept A is
conjugate base of concept B, then B is conjugate acid of A. A
similar situation exists for the two relationships is tautomer of
and is enantiomer of [
The types of errors that the experts were looking for included both errors of commission and errors of omission. Examples of the former are incorrect hierarchical relationship, incorrect lateral relationship, and incorrect relationship target. Examples of the latter are missing hierarchical relationship and missing lateral relationship.
A random sample of 400 concepts (0.6%) was selected from the 61,895 concepts in ChEBI, February 2016 release. Out of these 400 concepts, 388 (97%) are from the chemical entity hierarchy, 11 are from the role hierarchy, and one is from the subatomic particle hierarchy. In the following, we often refer to a ChEBI concept using its name along with its unique ChEBI id, written, for example, as “CHEBI: 31900” (which is the concept with the name Neticonazole hydrochloride).
Two of the authors (HY and LC), both subject-domain
experts in chemistry and experienced in ontology QA, carried
out the individual QA analyses on the entire sample of
concepts and then produced a consensus report on the errors
discovered. Out of the 400 concepts, the two subject-domain
experts agreed on the errors reported for 167 concepts (41.8%).
The margin of error at the 95% confidence level for a 400
concept sample from a population of 61,895 concepts is 4.9%
[
There were 122 (30.5% = 122 / 400) concepts that exhibited errors of omission. Of these, 121 concepts were found to be missing hierarchical relationships, and only one concept, fatty acid anion 4:0 (CHEBI: 78115), was reported to be missing the relationship is conjugate base of.
Table 1 shows the number and percentage of concepts with errors of commission. For example, 36 concepts (9%) in the sample were found to have incorrect is a relationships. Note that some concepts may have multiple kinds of errors. For example, there are 17 concepts that were reported to have both errors of commission and omission.
Table 2 and Table 3 list examples of erroneous concepts with errors of commission and omission, respectively, along with their corresponding suggested corrections and the reason for the error. For example, in Table 2 (Row 3), we see that Neticonazole hydrochloride (CHEBI: 31900) was originally modeled as is a hydrochloride; instead, the modeling should be has part because the mixture contains hydrochloride.
This is the most common error, with 30.25% of the concepts exhibiting it. For example, there are 23 benzenecontaining compounds in the sample that should be classified as a benzenoid aromatic compound (CHEBI: 33836). Included among these is 8-hydroxy-3-chlorodibenzofuran (CHEBI: 79743).
This is the second most common error (7%). The correct charge difference between an acid and its conjugate base is 1, and the acid is 1 charge higher. This is because the acid has one extra proton (H+) compared to its conjugate base. As indicated in the equation HA A− + H+, HA is the conjugate acid of A−, and A− is the conjugate base of HA. The only difference between HA and A− is one proton; thus, HA is 1 charge higher than A−. For example, ChEBI concept 1-(2-carboxyphenylamino)-1-deoxy-D-ribulose 5phosphate (CHEBI: 29112) has as its conjugate base 1-(2carboxylatophenylamino)-1-deoxy-D-ribulose 5-phosphate (3−) (CHEBI: 58613). However, its conjugate base charge should be 1−, not 3−, since only one proton is removed from the acid, not three protons, as shown in its structure in Table 5.
Piperidine (CHEBI: 18049) is classified under Brønsted acid (CHEBI: 39141), but in fact it is should be classified as a Brønsted base (CHEBI: 39142). This is because an amine is a proton acceptor, not a donor, and it acts as a base not as an acid. Another more common occurrence of this kind of error is seen for 14 erroneous concepts that are classified as some class “A,” but, in fact, do not have chemical structure A. For example, in Fig. 1 showing the chemical structure of
contain the following chemical groups carboxylic ester (CHEBI: 33308), carbonyl compound (CHEBI: 36586), and ester (CHEBI: 35701). However, it is classified as a carboxylic ester, carbonyl compound, and ester in ChEBI.
There are 13 concepts reported with incorrect numbers of cyclic units. For example, buspirone hydrochloride (CHEBI: 3224) is classified as an organic heteromonocyclic compound (CHEBI: 25693) that has one cyclic structure. However, from the structure seen in Fig. 2, we can see that this concept contains four cyclic units. Similar errors are seen in (+)-tephrosone (CHEBI: 66201), 3beta,13-Dihydroxy-16-(hydroxymethylene)-13,17-seco-5alpha-androstan-17-oic acid, delta-lactone (CHEBI: 79677), c[G(2',5')pA(3',5')p] (CHEBI: 75947), diazoline (CHEBI: 53123), dipyridodiazepine (CHEBI: 63667), pyrazolopyridazine (CHEBI: 48383), etc.
Fig. 2. Structure of buspirone hydrochloride (CHEBI: 3224)
There are seven concepts classified as primary amide (CHEBI: 33256). However, they should be classified as secondary amide (CHEBI: 33257). For example, from Fig. 3, we can clearly see that Arachidonoyl dopamine (CHEBI: 31231) is a secondary amide (nitrogen group connected two carbon atoms), while it is denoted as a primary amide. Similar errors are seen in beta-D-glucosyl(1<->1')-N-eico-sanoylsphinganine (CHEBI: 84703), bistratamide I (CHEBI: 65508), N-(2 hydroxyhexacosanoyl)phyto-sphingosine (CHEBI: 64958), N-(3oxohexanoyl)homo-serine lactone (CHEBI: 29640), N-(2hydroxy-docosanoyl)eicosasphinganine (CHEBI: 66983), and N(4)-acetylcytidine (CHEBI: 70989).
As an example, the structure of diacylglycerol 38:7 (CHEBI: 86986) does not match with its name. Its name indicates that there are two esters, while its structure, seen in Fig. 4, has three R groups, meaning a triacylglycerol, not a diacylglycerol.
In this study, we discovered an error-rate of 42% in a
random sample of ChEBI concepts. Among these, 15.5%
suffered from severe errors of commission, e.g., incorrect
parents and incorrect relationship targets. The remaining
30.5% exhibited errors of omission. (Some concepts had
both kinds of errors.) One needs to compare this finding to
the reality in other ontologies of a similar caliber. One
ontology for which data of this kind exist is SNOMED CT
[
Let us note that ChEBI employs a star status (rating) system to indicate the level of annotation applied to a concept by the ChEBI curatorial team. A concept manually annotated by the team has a “3-star” status. A concept manually annotated by a third party has a “2-star” status. A preliminary concept loaded automatically from a data source but not yet manually annotated is designated with a “1-star” status. We looked at the distribution of erroneous concepts according to their star status (see Table 6). Out of the 400 concepts that we analyzed, 292 concepts had a 3-star status, and 116 of those concepts (39.7%) were deemed to be erroneous. Among the 1-star concepts, 3.3% were erroneous (Table 6). And 64.1% of the 2-star concepts were erroneous. Collectively, the non-3-star concepts exhibited an error-rate of 47.2% (51 out of 108). So, while it was not surprising that the 3-star concepts showed an overall lower error rate, they still contributed significantly to the error findings with a rate of nearly 40%.
In the present study, we assessed the frequency of errors
of commission and omission in a random sample of
concepts from ChEBI. During our original design and
analysis, we postulated that concepts with higher numbers
of parents would exhibit higher error rates. This postulation
was based on a recurring, substantiated theme in our
previous ontological QA research that more complex
concepts are prone to exhibit higher error rates than
concepts in random control samples. Concepts with multiple
parents are indeed more complex than concepts with a
single parent due to multiple inheritance of properties and
convergence of definitional paths. For example, in our QA
research on the CORE problem list of SNOMED CT, we
indeed have shown that the expected error rate increases
with the number of parents [
However, as seen in Fig. 5—and contrary to our expectations—our postulation was wrong. In fact, our findings show that the error rate is inversely proportional with respect to the number of parents. For example, 12,128 concepts in ChEBI have two parents. From these, 97 were chosen for our random sample; 33 of them (34.0%) were found to be erroneous. Of the 38 three-parent concepts in the sample, 12 (31.6%) were erroneous. Further reductions in the error rates were seen for four-parent (22.2%) and fiveparent concepts (14.3%).
ChEBI is user driven, and user requests can be made via
the ChEBI submission tool [
The role of ChEBI in chemistry applications is significant. Therefore, findings of problems at this high level are of major concern. During the past 20 years, SNOMED CT (in which we have seen lower error rates in random concept samples) has been managed by a variety of large professional organizations, such as the College of American Pathologists (CAP), the National Library of Medicine (NLM), and the IHTSDO. Unfortunately, ChEBI does not have the same level of resources that have been available for the maintenance of SNOMED CT. Hence a creative solution to handle the QA chores of ChEBI is needed utilizing ChEBI’s curatorial board. It certainly makes sense, as a start, for the curatorial board of ChEBI to conduct a follow-up study of an even larger sample of concepts than the one used in the present study to further assess the error rates in ChEBI for errors of omission and commission.
In future work, ChEBI’s curatorial board may want to
identify criteria that can be used in locating subsets of
concepts that are more likely to be erroneous. As noted, the
number of parents as such a criterion did not prove useful,
but maybe there are others that will. Employing useful
methodologies to help automate aspects of QA efforts
should increase the yield of corrections with respect to the
curators’ expended time. Another potential way to measure
the complexity of concepts is by their number of
relationships. In a future study, we will test to see if ChEBI
concepts with larger numbers of relationships have higher
error rates than concepts with fewer relationships. In a
recent study, this was shown to be true with statistical
significance for the concepts in the Biological Process
hierarchy of the National Cancer Institute thesaurus (NCIt)
[
In this paper, we reported on a quality-assurance (QA) study that was carried out on a sample of ChEBI concepts by two chemistry subject-domain experts. The results revealed that quite a few ChEBI concepts suffer from some kinds of modeling problems. Our consensus report found that 15.5% of the concepts from our sample exhibited severe errors of commission. Particularly prevalent were errors of the type “incorrect and missing chemical classification” and “incorrect charge differences between conjugate acids and conjugate bases.” These findings are particularly troubling taking into account the importance of ChEBI and the many applications dependent on it. In general, it appears that the QA processes in place for ChEBI could use further refinement. For example, a targeted effort to review all charge differences between conjugate acids and conjugate bases in ChEBI seems warranted.
Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA190779. The content is solely the responsibility of the authors and does not necessarily represent the views of the National Institutes of Health.