-We present a Machine Learning methodology using a Convolutional Neural Network to perform a specific case of an ontology Quality Assurance, namely discovery of missing IS-A relationships for Neoplasm concepts in the National Cancer Institute Thesaurus (NCIt). The training step checking all “uncles” of a concept is computationally intensive. To shorten the time and to improve the accuracy, we define a restricted methodology to check only uncles that are similar to each current concept. The restricted technique yields higher classification recall (compared to the unrestricted one) when testing against known errors found by domain experts who manually reviewed Neoplasm concepts in a prior study. The results are encouraging and provide impetus for further improvements to our technique.
INTRODUCTION
Ontologies play a major role in enabling precise
communications and in support of healthcare applications, e.g.
EHR systems. Many ontologies are large and complex. For
example, the National Cancer Institute Thesaurus (NCIt) [
However, QA resources for ontologies are typically scarce, while QA tasks are labor-intensive and time-consuming. Therefore, automated or semi-automated techniques that can either help in auditing an ontology or narrow down the places where to look for errors, are highly desired. Missing parent/child errors are particularly interesting to ontology curators, as the IS-A links are the backbone structure of an ontology, facilitating the inheritance of lateral relationships (called roles in NCIt).
Machine Learning (ML) has been proven successful in
many fields, e.g., knowledge mining. ML was previously used
in knowledge enrichment for ontologies [
In an unpublished study, we trained a Convolutional Neural
Network (CNN) deep learning model to insert new concepts
into the SNOMED CT ontology, i.e., an enrichment problem.
In the present work, we train a CNN deep learning model to
find missing parent/child errors in the Neoplasm subhierarchy
of NCIt. The vector representations of concepts are obtained
from an unsupervised neural network language model. The
model is evaluated by its classification recall on an unseen
dataset. We check the model’s classification recall by testing
against 18 missing parent/child errors found by domain experts
in a prior study [
In previous research we have introduced Abstraction
Networks (AbNs) [
To accelerate the processing and improve recall, we modify the CNN methodology to limit its consideration, for each concept, to the similar concepts of its area (in our formal sense of area). The modified, restricted methodology achieves 0.81 recall on the unseen testing data. It performs 50% better than the unrestricted methodology on the 18 known errors in terms of recall. The results for detecting missing IS-A links are not yet strong enough. However, the performance in recognition of known errors is encouraging and supports further improvement of our methodology in respect to CNNs and the use of AbNs.
II.
BACKGROUND A. Doc2vec
Numeric representation of variable-length texts, ranging
from sentences to documents is a challenging task. Doc2vec, or
Paragraph Vectors [
Convolutional Neural Networks (CNN), initially invented
for image recognition, have been widely used for various
applications, including vision, speech recognition, and
language translation. CNN models have also been successfully
applied to solve various Natural Language Processing (NLP)
problems such as search query retrieval [
NCIt is published monthly by the National Cancer Institute (NCI) in OWL and flat file formats. It is a cancer reference terminology that is widely used. It covers cancer-related terminology in various fields, e.g., clinical care and translational and basic research. Concepts are linked to other concepts (parent concepts) in the same hierarchy by IS-A relationships. A concept may have multiple parent concepts. The semantics of concepts are defined by lateral relationships (called “roles”), e.g. Disease Has Associated Anatomic Site.
Due to the NCI’s cancer focus, the Neoplasm subhierarchy of NCIt is composed of 9,955 concepts. It is a core component of the Disease, Disorder or Finding, the largest hierarchy with 35,081 concepts, and is modeled with more detail, compared to non-neoplasm concepts in the hierarchy.
An area taxonomy [
Fig. 1(a) is an excerpt of 12 Neoplasm concepts from NCIt. Concepts are represented as rounded-corner boxes and the arrows denote IS-A relationships. Concepts with the same set of role types are enclosed within a colored dashed rectangle. For example, both Benign Neoplasm and Tumorlet have the two role types Disease Excludes Abnormal Cell and Disease Has Abnormal Cell, they reside in the left green dashed rectangle. Fig. 1(b) shows the area taxonomy for Fig. 1(a). Each colored, dashed rectangle in Fig. 1(a) becomes an area with the same color in Fig. 1(b). An area is labeled by its role type set and the number of concepts it summarizes. Areas with the same number of role types have the same color. For example, there are two areas colored in green, since both have two role types. Skin Neoplasm is the root concept of the red area and its parent Neoplasm by Site is in the grey area. Hence, there is a child-of relationship from the red area to the grey area, denoted as a bold arrow in Fig. 1(b).
We describe two methodologies, the unrestricted methodology and the refined restricted methodology. The ML training problem is viewed as a binary classification task: given a concept pair, we classify it into a positive category (there is an IS-A link) or a negative category (there is no IS-A link). We train a Convolutional Neural Network (CNN) model to solve this classification problem.
Unrestricted Methodology: To train a CNN model to yield high precision, we must carefully choose the training data for both categories. The source of training samples for the positive category is in the IS-A hierarchy of the ontology. The challenge is in the choice for the negative samples as we cannot use the full set of unconnected pairs. For the Neoplasm subhierarchy of NCIt with 9955 concepts, the size of this set is 99,075,537 (= 9955*9954 – 16533). We subtract 16533 existing IS-A link pairs from the potential missing parent/child errors. Training pairs should not be chosen randomly. We need to choose pairs where there is a reasonable likelihood for an IS-A link, not pairs that obviously have no taxonomic relation. For example, a body part concept and a drug concept are not related by an IS-A relationship and would be a bad training pair. Negative training samples should be near misses, close to the “hyperplane of separation” in SVM terms.
To address these two problems of magnitude and recall in the ML training, we limit the negative samples for the unrestricted methodology to only “uncle - nephew” pairs for only near misses. That is, connections between a concept and a sibling of its parent. By only choosing “ uncle - nephew” pairs, we guide the model to learn the underlying features used to distinguish IS-A-connected concept pairs and similarly positioned concept pairs that have a high potential to have secondary IS-A links but are not connected by IS-A links. There are total 37,147 such “uncle - nephew” pairs in the Neoplasm subhierarchy of NCIt.
The unrestricted model must consider all uncles of a concept (that are not connected to that concepts by an IS-A link). Due to the size of the Neoplasm subhierarchy, the number of uncles is large. For example, there are 24, 15 and 15 concepts with 10, 11 and 12 children, respectively and the maximum number of children of a concept is 60. Each grandchild of a concept with 15 children has 14 uncles. If the average number of children of each sibling is 5, then there are 70 concepts with 14 uncles each. Applying the proposed technique to select negative samples from the whole hierarchy results in a large number of “uncle - nephew” pairs. This is computationally expensive for training the model and leads to low accuracy by distracting the model from learning subtle features that distinguish between IS-A and “uncle - nephew” links within similar groups of concepts. In other words, not all “uncle – nephew” pairs are equally useful for training purposes. We need pairs that are similar to existing IS-Aconnected pairs, yet are not themselves IS-A-connected.
Additionally, many machine learning models work best with balanced training sets. The number of positive and negative samples should be approximately equal. The number of positive training instances in our problem domain is given and fixed. The number of potential negative training samples is much larger. Thus a cogent way has to be used to select a number of negative training samples that is closer to the number of positive training samples.
The Restricted Methodology: To cope with these problems,
we introduced the restricted approach. The restricted approach
limits the number of negative samples by only choosing a
subset of a concept’s uncles, which are structurally similar to
the investigated concept. To provide such a “closely related”
subset we partition the Neoplasm subhierarchy into sets of
concepts of similar structure. In doing this we are availing
ourselves of a powerful mechanism that derives an area
taxonomy from an ontology [
In the area taxonomy each cluster of concepts constitutes
an area. Due to the high average number of roles per concept in
the Neoplasm subhierarchy, the number of areas is large, and
the average size of an area is small. Note that AbNs are
automatically derived from the ontology so the limitation
process is automatic [
Since all uncles in an area will have exactly the same roles as the current concept, they will be similar to it. Thus the recall of the CNN training model is expected to be higher than for the unrestricted model trained with all uncles of a concept, many of which are not similar to the current concept.
The following description is related to both methodologies. Overall, our methodology comprises following four steps:
The CNN model requires its input in the format of fixedlength feature vectors. Thus, before sending concept pairs for training, we need to transform each concept into its corresponding vector representation with fixed length.
The Paragraph Vector (Doc2vec) framework introduced by
[
To derive the vector representation of a single concept, we need text “descriptions” of the concept. We recast a concept into a document such that it preserves hierarchical and partial semantic information of the concept:
The document of a concept contains the concept ID, the name(s) of its ancestor(s), the name(s) of the concept itself, the name(s) of its child(ren) and the names of its grandchild(ren), if they exist. In this way, the document implicitly maintains the hierarchical relationships of the ontology.
For example, the document representation of the concept
distributed vector representation maintains the most important hierarchical relationship semantics.
The document vectors are derived using the Distributed
Memory version of Paragraph Vector (PV-DM) [
The positive samples are directly extracted from the hierarchy as all the concept pairs connected via an IS-A link.
is a positive sample, because Malignant Nipple Neoplasm is a Nipple Neoplasm (Fig. 2). As mentioned above, there are 16,533 positive samples in the Neoplasm subhierarchy. We randomly picked 2,000 positive samples for testing. The remaining 14,533 (=16,533-2,000) samples are used in a ratio of 80% for training and 20% validation. The positive samples are treated equally for both models.
For the restricted model, there are 10,574 potential pairs where both concepts of each pair are from the same area. We randomly picked 2,000 for testing. As noted above, there are 37,147 “uncle – nephew” negative sample pairs in the hierarchy. However, for the unrestricted model, we use the same 2000 negative pairs, used for the restricted model. This is done to enable performance comparison between the two models. Similar to the way we handle the positive samples, the remaining 8,574 (=10,574-2,000) samples for the restricted model and the remaining 35,147 (=37,147-2,000) for the unrestricted model are divided to 80% vs. 20% ratio for training and validation, respectively.
In addition, we down-sampled the negative samples for the unrestricted model and up-sampled for the restricted model to 14,533 samples, in order to balance the number of samples for both categories, as customary in Machine Learning.
We trained a CNN model with 4 convolution layers on top of vectors derived from the Neoplasm subhierarchy of NCIt via Doc2vec. The CNN model architecture is shown in Fig. 3. The input to the CNN model is two 128x1 dimension vectors and the output is a 2x1 dimension vector.
Some structural details of this model are summarized as follows: • • •
There are four convolution layers, each followed by a max pooling layer. This choice was informed by previous research. The first convolution layer has 18 filters with kernel size =1. The filter number doubles with the increase of convolution layers. We use stride =1, meaning we slide the filters one number (position) at a time over the input. The pooling size is 2 for all max pooling layers.
used in every convolution layer, because it has proven successful in recent research projects. This corresponds to a “rectifier function” from electrical engineering, blocking the negative half-wave and letting the positive half-wave pass through one-to-one.
Traditionally, machine learning models are tested with kfold cross validation. Thus, all known data is partitioned into k folds, the model is trained with k-1 folds and tested with the remaining fold. This process is repeated k times, with resulting precision, recall and F1 values averaged. We have augmented this testing by human expert quality assurance results.
In a previous study [
IV.
The testing recall is 0.75 and 0.81 for the unrestricted and restricted models, respectively. Fig. 4 (a) and (b) show the Receiver Operating Characteristic (ROC) curves of the testing performance for the unrestricted and restricted models, respectively. The AUC (area under the curve) scores, as the measure of test accuracy, are 0.84 and 0.90, respectively.
The restricted model detected 10 out of the 18 errors that
domain experts found [
Each model is trained with 2000 epochs, with batch size = 2000. We recorded the duration of the training. With the same computer hardware configuration, training the unrestricted and restricted models took 1116 and 1110 seconds, respectively.
To the best of our knowledge, this is the first published attempt to use Machine Learning (ML) for QA of ontologies. Such technique can prepare a subset of pairs of concepts as candidates for missing IS-A link omission errors, optimizing the use of scarce QA resources.
In this paper, we discussed two ML approaches. The unrestricted model utilizes all uncle concepts of the processed concept. The restricted model is further taking advantage of the area taxonomy of the ontology to utilize only uncles in the area of the processed concept. Both ideas save processing time while improving the accuracy, by training unconnected pairs of concepts similar to the original IS-A links of the processed concept. The uncles are similarly positioned as siblings of the parent of the processed concept. The uncles within the area share the same roles as the processed concept.
Table II compares the performance of the two models. As
can be expected, the restricted model utilizing training with
similar concepts achieves higher performance. We evaluated
the results of the two ML models based on a list of errors that
domain experts found in a previous study [
From Table II we can see that the restricted model performs
6% better than the unrestricted model in recall. The area under
the curve in Fig. 4(b) is 6% larger than in Fig. 4(a), reflecting a
better classification of the restricted model than the unrestricted
model. The document vectors used in this study were derived
using the Distributed Memory version of Paragraph Vector
(PV-DM) that works well for most tasks, as stated in the
original paper [
The recall obtained is not high enough for reliable QA for missing IS-A links. For example, out of a random subset of 20 suggested errors, a domain expert (GE) confirmed only one error pair, Hair Follicle Neoplasm missing the parent Dermal Neoplasm, since hair follicles reside in the dermal layer of the skin. In the NCIt, Dermal Neoplasm has 13 children, representing a mix, based on cell origin as well as malignancy status. Currently, Hair Follicle Neoplasm has Skin Appendage
only a very indirect relationship to the dermis through the
the target value and the Anatomic Structure Is Physical Part Of role of Hair Follicle with the target value of Dermis. Adding Dermal Neoplasm as a parent, or even possibly replacing Skin Appendage Neoplasm with it, might be better.
A higher recall will imply fewer suggested errors with a higher percentage of confirmed errors by domain experts. In future research, we will further explore the properties of the two models as well as properties of ML processing in an effort to best fine-tune and utilize both models, to increase the recall.
LIMITATIONS
We presented a supervised training technique, therefore “annotated corpora” are required for its applicability beyond finding missing IS-A relationships in the NCI neoplasm taxonomy. The results are also limited, because the models are not trained on more general data for more general problems.
VII.
CONCLUSION
We explored whether ML methods can be applied to the task of ontological QA, in particular whether ML can help in detecting missing IS-A links in an ontology. Two models are presented. Our application of ML to the Neoplasm subhierarchy of NCIt demonstrated that the restricted model performs better than the unrestricted one. However, the performance of the restricted model is not yet sufficient for QA. In future research, we will explore improvements in ML processing and more accurate restrictions for concepts in the training stage to improve performance of QA of ontologies.
Research reported in this publication was partially supported by the National Cancer Institute of the National Institutes of Health under Award Number R01CA190779. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.