A bottleneck problem in detecting knowledge nodes and their relations is how to extract accurately and correctly and codify the complex knowledge assertions from full-text documents (human intelligence) into the format of “machine intelligence” (computer-processable knowledge assertions). This paper reports a preliminary study that aims at this bottleneck problem by starting from the fundamentals of KR-representing knowledge from full-text documents by using knowledge node and relation recognition methods and tools. We collected data from full-text biomedical research publications and used manual and automatic tools to investigate the strengths and limitations of these methods. The findings show that MetaMap did a better job in detecting concepts from texts while SemRep is capable of extract relations between k-nodes. The paper presents the findings from the perspectives of degree of abstraction, types of k-nodes and relations, and linguistic structures and the evaluation results using the BLEU and cosine similarity measures.
Document content is typically represented by keywords and/or terms from knowledge
organization systems (KOS). Although numerous algorithms and models, for example,
the vector space model
One way to address the challenges facing term-based document representation is to shift the representation approach from terms to knowledge networks consisting of nodes and relations. This means a transition from term-centric representation to concept-relation representation: terms as knowledge nodes (or k-nodes) are linked by relations to form a knowledge network. For example, influenza is a single concept representing a kind of disease, H1N1 as a kind of virus is a single concept, and the two concepts are related because H1N1 virus causes influenza, which may be written as a knowledge assertation (H1N1, causes, influenza). If we analyze it further, we can find that this assertion implies two separate assertions: (H1N1, IS-A, virus) and (influenza, IS-A, disease), hence the complete assertion may be expressed as [(H1N1, IS-A, virus), causes, (influenza, IS-A, disease)]. These assertions (or factual knowledge) can be generalized in a triple format (concept1, relation, concept2) or (subject, predicate, object), whereas concepts and relations may be complex and nested. This example is used to demonstrate a need to reexamine the nature of indexing or document content representation from an epistemological approach to enable richer, more semantic representations.
Representing knowledge assertions has its roots in artificial intelligence (AI). In the
short history of AI, knowledge representation formats such as predicate logic, frame
systems, semantic networks, and rule systems from the official production system
family have been popular
A bottleneck problem in this paradigmatic shift lies in detecting k-nodes and their relations accurately and correctly and codifying complex knowledge assertions from full-text documents (human intelligence) into the format of “machine intelligence” (computer-processable knowledge assertions). This paper explores the epistemological and linguistic aspects of this bottleneck problem by starting from the fundamentals of KR—representing knowledge from full-text documents by using k-node and relation recognition methods and tools.
Detecting knowledge nodes and relations in full-text documents automatically and
representing them in some degree of formality is similar in many ways to knowledge
representation in AI. What is knowledge representation then? The notion of KR implies
five distinct roles according to
Research in KR in the information science field appears to have followed two
different but intertwined approaches, both having a great deal to do with the data sources
under study. The first approach uses KOS as the data source to transform the concepts
defined in KOS into knowledge maps or graphs. In order to make such transformation
possible, a key task is to establish relations between concepts and the existing concept
scope and linguistic relations become handy in defining the transformation data models,
as seen in the Simple Knowledge Organization System (SKOS)
Knowledge representation also occurs in transforming legacy data models (e.g., relational database, XML) into RDF or Web Ontology Language (OWL) format or integrating multiple linked data repositories to build a specialized KR base. In reviewing 14 articles that cited any of the three significant linked data projects—Bio2RDF, Open PHACTS, and/or EBI RDF, Barros and Couto (2016) conclude that RDF technologies have “a strong impact on how the Life and Health Sciences community is storing, integrating and sharing data and knowledge” (p. 182). Although automatic methods such as text mining have been used to support semantic annotations to common ontologies, human intervention is still required in most cases (Barros & Couto, 2016,).
The goal of transformation of existing thesauri or controlled vocabularies into RDF encoded format is to codify the concepts and relations in more formal and better structured format for efficient computation. This approach, however, does not connect the knowledge existing in documents and research data to KOS. In other words, the fact that controlled vocabularies are transformed into RDF format only builds a semantic infrastructure to support the representation of knowledge existing in documents and data. The work of connecting the knowledge in documents and data to KOS is what we usually refer to as indexing, text categorization, document classification, or document representation. In this paper, we take document content representation a step further to formalize such representation in the form of (subject, predicate, object), that is, representing not only knowledge nodes but also the relations among the nodes.
Representing knowledge in documents has been studied extensively and generated
a vast body of publications. Popular models include the vector space model
Recognizing medical concepts and their semantic relations is a fundamental task in
biomedical informatics. Concept and relation recognition can support a range of tasks
such as concept-based text retrieval
MetaMap
Built on the concept recognizers, another suite of tools was developed for identifying
semantic relations between concepts. SemRep
The main objective of this project is to gain insights into the methods and issues in
knowledge node and relation recognition. Hence the data collection was not intended
to be comprehensive nor representative, but rather, focused on the methodological
aspect to gather information for larger scale investigation. This project continues the work
reported in
The source documents were retrieved from PubMed by using query terms “precision
medicine” in combination with break cancer, diabetes, and oncology. The selection of
articles was described in
Example sentence #1: “Protein gene products that have direct roles in driving the biology and clinical behavior of cancer cells are potential targets for the development of novel therapeutics” is annotated in Table 1:
Node A relation Node B protein gene product drives biology behavior of cancer cell protein gene product drives clinical behavior of cancer cell protein gene product is-target-of therapeutics
Example sentence #2: “Unlike most pathologic testing, which serves as an adjunct to establishing a diagnosis, the results of HER2 testing stand alone in determining which patients are likely to respond to trastuzumab, a monoclonal antibody against HER2” is annotated in Table 2:
The relation “have direct roles in driving” in example sentence #1 was simplified as
“drives”, and the pattern of k-node and relations in Table 1 can be generalized further
as “factor-A drives behavior of cancer cell.” In identifying k-nodes we tried to be
faithful to the original concepts as much as possible, and at meantime, we also needed to
derive and/or extract k-nodes that were general enough to be meaningful in as many
situations as possible. This is particularly true for terms representing relations. In
addition, linguistic patterns in the sentences were also noted to help generating potential
NLP rules for automatic k-node detection. The manually annotated k-nodes were then
matched with concepts in the Unified Medical Language System (UMLS)
Metathesaurus
Two tools were used to generate k-nodes and relations automatically. The first one
is MetaMap, which was developed at NLM for recognizing UMLS concepts in text.
The strength of MetaMap lies in recognizing concepts from texts and matching them
with terms in UMLS. It does not perform, however, the task of detecting relations
between concepts from text. SemRep was selected because of its ability to extract
threepart propositions (or semantic predications) from biomedical texts
To obtain comparative data for the manually annotated k-nodes and relations, we ran
MetaMap and SemRep on the same texts. The resulting k-nodes and relations were
evaluated to find strengths and weaknesses of automatic k-node and relation detection
by using the tools. These steps generated a number of datasets: 1) manually annotated
k-nodes with corresponding MetaMap results, 2) k-nodes generated from MetaMap and
matching UMLS concepts, 3) relations generated manually and by SemRep, and 4)
evaluation scores using two algorithms: Bilingual Evaluation Understudy (BLEU) and
cosine similarity for evaluating the similarity between manual and automatic k-node
detection. BLEU was originally designed to evaluate a generated sentence to a
reference sentence, with 1.0 as a perfect match score and 0 representing a mismatch
Overall, the results show some clear differences in the types of k-nodes and relations between manual annotation and automatic detection. The differences are more visible when k-nodes and relations are represented in the three-part semantic predicate format. We observed three areas of differences in k-node and relation recognition: degree of abstraction or generalization, types of k-nodes and relations, and linguistic structures. 4.1
Semantically, the k-nodes captured by each method have a high similarity. Regardless how each concept is expressed in language, keywords for main concepts are present across three sets of results. A closer examination of resulting concepts shows that there are varying degree of abstraction or generalization between three methods. It is easy for human intelligence to determine that “monoclonal antibody against HER2” and “time from tissue removal to tissue fixation” are concepts and separating the words in any of the two phrases would cause information loss (Table 3). Among the three methods, MetaMap seems to be the better one for recognizing complex concepts involving long phrases. SemRep breaks the sentences into simple concepts with one or two words, while manual results appear to be in between the two automatic tools. In this sense, manual method and MetaMap can handle better complex concepts while SemRep generates simple k-nodes.
logic testing, which serves as an adjunct to establishing a diagnosis, the results of HER2 testing stand alone in determining which patients are likely to respond to trastuzumab, a monoclonal antibody against HER2.
At present, several preanalytic factors, including the time from tissue removal to tissue fixation, are underappreciated as important variables that have the potential to negatively impact the consistency and reliability of HER2 testing.
the breast is a leading cause of cancer morbidity and mortality among women world
wide.
A major challenge faced by clinicians treating patients with breast cancer is how to best assess patient outcomes and predict the clinical course of the disease so that the most appropriate treatment regimen can be identified.
Another way to look at the degree of abstraction is through the relations detected
from the text. Table 4 presents the relations identified by SemRep and manual
annotation. While the manual work resulted in a much larger number and variety of relations
than SemRep did, SemRep has its own predefined list of relations and many of them
can be found in the relation types defined by the UMLS semantic network
In the process of manual annotation, we noticed some patterns of k-node and relation structures.
1. Simple k-node relations: two simple k-nodes are connected by a direct relation in the form of a single verb, which may be expressed as AàB, e.g.: (amplification_of_HER2_gene, promotes, receptor_activation) (tumor, harbors, HER2_molecular_alteration)
Exact match affects
is-a is-method-for is-part-of treats uses
Relations from manual annotation
Similar/Partial match allows, improves, impacts,
promotes provides, controls is a kind of, exists, is equivalent to, is a prototype for, is given as
is-for correlates is associated
with expands Causes, makes, leads to, promotes, drives, determines improves is measured by, is-testedby, measures, is-in-context is a factor of, has-attribute, has condition of
transmits mediates, has-concordance with, targets
regulates is-used-for, is-used-with
Compound k-node relations: it is common in the text used for this study that a k-node is related to more than one k-node that may have the same or different relations. We use the expression Aà(B1…Bn) to represent a one-to-many pattern. For example, in a sentence “Therefore, over-expression of the receptor has a direct role in mediating the biologic and clinical behavior of HER2positive tumor cells by driving their proliferation and survival”, we derived: (overexpression_of_receptor, mediates, biology_behavior_of_HER2positive_tumor_cells) (overexpression_of_receptor, mediates, clinical_behavior_HER2-positive_tumor_cells) (overexpression_of_receptor, drives, proliferation_of_tumor_cells) (overexpression_of_receptor, drives, survival_of_tumor_cells) Based on the expression Aà(B1, …, Bn), the above predicates can be written together if they share the same relation: (overexpression_of_receptor, mediates, (biology_behavior_of_HER2positive_tumor_cells, clinical_behavior_HER2-positive_tumor_cells)) (overexpression_of_receptor, drives, (proliferation_of_tumor_cells, survival_of_tumor_cells)) This pattern is only applicable to the situation in which one concept is related to multiple concepts by the same relation. Once the relation changes, the knode-relation triple also changes to become a new one.
Complex k-node relations: sometimes a sentence contains multiple k-nodes and the relations among them are not simple nor direct. More often than not, these multiple k-nodes are chained together by “bridge” k-nodes. We use the expression Aà(BàC) to represent such “chain relations” where k-node A is related to C through bridge k-node B. For example, the sentence “Unlike most pathologic testing, which serves as an adjunct to establishing a diagnosis, the results of HER2 testing stand alone in determining which patients are likely to respond to trastuzumab, a monoclonal antibody against HER2” contains a complex k-node and relation and a simple k-node and relation proposition: (HER2_testing, determines, (patient, responds-to, trastuzumab)) (trastuzumab, is-a, monoclonal antibody against HER2) 4.3
While we kept relations between k-nodes in the form of verbs, the linguistic structures for candidate k-nodes appeared in a wide variety of patterns. To decide what may be meaningful and appropriate k-nodes, we needed to address two challenges. First, identifying a k-node from a long phrase (in which may contain a clause) often means to make a decision on where the cut-off point is. For instance, “Protein gene products that have direct roles in driving the biology and clinical behavior of cancer cells” is a long phrase that contains a clause modifying the k-node “protein gene product”. It would not be feasible to take the whole phrase plus the clause as the k-node but cutting off the clause would cause an important information loss.
Second, as a result of the first challenge, there are situations in which candidate relation terms in non-verb or clause format need to be transformed into verbs or verb phrases to avoid the information loss (see Example 1 and Table 1 in the Methods section). Such non-verb-to-verb transformation often results in multiple k-node-relationk-node propositions to represent the complex concepts.
In the process of manual annotation, we generalized some linguistic patterns for knodes: • Simple k-nodes: nouns that may be mapped directly to concepts in UMLS; • Compound k-nodes: noun phrases that consist of two or more nouns and each noun may find a matching concept in UMLS; • Complex k-nodes: this type of k-nodes often appears in the form of noun/noun phrase + preposition + noun/noun phrase format, for example, “dysregulation of gene”, “therapeutic target in clinical oncology”, and “sensitivity to cytotoxic drug”. 5
To compare the manual annotation results to those generated by automatic tools, we used MetaMap and SemRep to extract k-nodes and relations and applied the BLEU and cosine similarity algorithms to calculate the similarity scores. The average similarity scores in Table 5 show that MetaMap and manual methods in k-node recognition has a higher degree of similarity compared to other two pairs. Figure 1 visualizes the comparisons between each pair of three methods. This is also illustrated by Figure 1 (a) where more k-nodes had higher scores for both measures than the other two pairs, as Figure 1 (b) and (c) show that the scores fluctuated widely and the scores from BLEU and cosine similarity methods were not as consistent as that in Figure 1 (a).
We also evaluated the similarity between the k-nodes generated by three methods and matching UMLS concepts in each result set. Table 6 shows the total numbers of UMLS terms for each comparison as well as the average scores. Overall, UMLS matching had much lower averages across three comparing pairs than the scores in Table 5. (a) Manual vs. MetaMap
(b) Manual vs. SemRep (b) MetaMap vs. SemRep It is worth pointing out that both BLEU and cosine similarity followed consistent patterns, that is, there was a large number of 0’s and 1’s in the scores. This suggests that a high number of k-nodes detected in the text did not have their counterparts in UMLS. This is consistent throughout the mapping between UMLS and each of the three k-node detection methods.
No matching for UMLS terms 0.227 0.204 5589 (c) Manual vs. MetaMap
(b) Manual vs. SemRep (c) MetaMap vs. SemRep Discussion and Conclusion
knowledge node and relation recognition from full-text documents is highly challenging, yet critically important in the big data era. This paper is an attempt to examine different approaches in k-node and relation detection and hope to be able to offer some insights into the strengths and limitations of manual and automatic methods. Due to the small size of the data, the findings from this study are far from conclusive in addressing the three questions raised at the end of methods section. Nevertheless, we can learn a few things from the findings.
First, each of the manual and automatic k-node and relation detection methods has different areas of strengths and limitations. Manual method can offer fine granularity for k-nodes and relations, but it is slow and heavily relies on human coder’s knowledge and analytical capability. MetaMap does a decent job in capturing k-nodes but lacks the capability for relation detection. SemRep can derive concepts and relations to put them in a three-part proposition format, but the k-nodes extracted by SemRep are essentially single words and the relations are strictly limited to what is available in UMLS Semantic Network. K-node and relation detection has much to be desired in the AI era.
Second, there is clearly a gap between effective knowledge representation from fulltext documents and the tools. Here the tools include not only software but also codified rules and knowledge assertions. Although knowledge representation and knowledge base research have achieved a great success (e.g., the Cyc knowledge base and DBpedia), representing knowledge from full-text and codifying it for applications remains a challenging research field. Having upper-level KOS can be useful for knowledge representation from full-text, but they can be useful only at a coarse level; deeper and more refined knowledge representation for full-text requires more research on natural language processing and computational knowledge organization systems.
Finally, automatic relation detection is perhaps the crown in knowledge representation because one must first identify the k-nodes before relations among the nodes can be determined. Knowledge assertions in three-part proposition form (or simply, triples) has its root in AI. This three-part proposition is the foundation of modern KR that has been actualized in semantic web technologies – RDF, Linked Data, and Web Ontology Language (OWL). KR combining with semantic technologies have promises to revolutionize information retrieval, presentation, and use, but such promises have to meet the prerequisite – knowledge codified and structured as the content infrastructure upon which innovations in information retrieval, presentation, and use can be built. Our work on k-node and relation detection is a step toward this vision.
Representing knowledge in full-text documents is not a new research field, but with technology advances (semantic web, graph database, deep learning, etc.), this “old” research field is being injected fresh possibilities. Our future study will explore other tools such as NOBLE Coder on larger text collections.