The output of the majority of NLP based informatics pipelines for structurization of free-text lack an ability to recover and convey implicit information, found in diagnostic reports. Such information is readily perceived and taken into account by a human reader as a contextual component. Here, we have developed a method to model contextual information in order to recover implicit relationships among structurized diagnostic entities. Our method enables structurization of contextual information into a cohesive and holistic representation of free-text diagnostic reports, which we call Knowledge Graphs. An expert assessment confirmed the capability of the method to correctly convey contextual information. The precision of matching of the semantical content of the free-text with the corresponding knowledge graphs was 0.92 and the recall was 0.84. The Fisher's exact test had odds ratio 19.7 and p-value of 2.2e-16. The intra-correlation coefficient (ICC) statistic that reflects the level of correlation and magnitude of agreement between domain experts was 0.818 (p-value of 0.99). These results indicate high level of agreement among all experts in the study.
Free-text sections of diagnostic reports contain descriptions of molecular data, microscopic findings from biopsy specimens, interpretations of laboratory values, clues for the identification of diseases, and data on disease surveillance.
In order to computationally analyze diagnostic reports, we need to convert free text
to a structured format. In this regard, natural language processing information
extraction techniques (NLP-IE) have been widely used to automatically extract knowledge
from free text via a structured relational triple format [
Several NLP-IE applications successfully convert free-text to relational triples.
Mausam et al. developed OLLIE, which extracted relational triples mediated by verbs, nouns
and adjectives [
Structured representation of text facilitates the use of computational models. Then,
computers can be used to mine for implicit relations between the data, to discover
patterns in the data, and to enable “semantic understanding and prompt retrieval” of
specific information from documents [
Note here that the majority of these current NLP-IE applications extract explicit
relations between entities that belong to the same sentence or clause [
Histologic sections show partial effacement of the lymph node with areas of nodular architecture [. . .]. Scattered Reed- Sternberg cells are present [. . .].
histologic sections show effacement of lymph node histologic sections show effacement histologic sections show partial effacement area of nodular architecture scattered Reed-Sternberg cells are present Figure 1Relational triples generated from sentences in a diagnostic report using NLP-IE applications
According to an expert pathologist, the focus of the diagnostic report is a “lymph node”. In this case, Reed-Sternberg cells should be considered in the context of that “lymph node”. However, since this fact was not expressed explicitly in the text, structurization algorithms would not convey it in their output.
As illustrated from the previous examples, context is essential component of
NLPIE. In biomedicine, the term “context” describes entities related to a biomedical
problem. Although the context of a whole text document is essential for the extraction of
implicit information, most NLP-IE applications instead focus on the context of a
sentence or a clause. For example, Mausam et al. use “attribution and clausal modifiers”
to extend a relational triple to a quadruple. The extra field provides contextual
information [
Histologic sections show partial effacement of the lymph node with areas of nodular architecture. The nodular areas are composed of a mixture of monomorphic small cells [. . .]. Immunostains with the appropriate controls are performed on block 1A [. . .]. Reed-Sternberg cells mark with weak nuclear positivity for PAX5 [. . .]. Reed-Sternberg cells are negative for CD20.
histologic sections show pwairtthiaalreefafsaocefmnoednutloafr layrmchpihtencotudree nodular areas are composed of mixture of monomorphic small cells immunostains are performed on block-1A Reed-Sternberg cells mark with weak nuclear positivity for PAX5 Reed-Sternberg cells are negative for CD20 Figure 2 Example of information extracted in the context of a diagnostic report using openIE applications.
The relational triples in Figure 2 demonstrate that the current NLP-IE system has
“awareness of the context of each extraction” [
In order to structurize implicit information, we need to model the context of a
diagnostic report in relational triple resource description framework RDF-like format,
which is a building block of a knowledge base (KB). Triples that share subject or object
induce a graph that we link using the n-ary relation schema according to the semantic
web [
The following section discusses in detail our methodological approach. 2
The informatics pipeline for modeling of contextual information is implemented as two independent processes (See Figure 3). The following sections describe these processes in detail.
To encode contextual information, we use a Diagnostic Practice Ontology (DPO). DPO consists of concepts and relationships that describe a specific diagnostic setting. For instance, DPO includes concepts related to the diagnostic process such as types of specimens, tissues and cells as well as various diagnostic tests. It also includes a hierarchy of personnel involved in a diagnostic process such as pathologists, residents, and laboratory staff. A structurized version of a diagnostic report consists of instantiations of DPO concepts and their relationships.
In a semi-automatic Background Process (top panel in Figure 3), diagnostic reports from a Laboratory Information System (LIS) are screened by a human expert to search for new contextual relationships that are not present in the DPO. For instance, headings of sections of diagnostic reports can represent contexts for concepts described in these sections. Another example is specific locations in tissue specimens that can serve as context for biological entities (e.g. germinal center is context for Reed-Stenberg cells).
To encode these relationships, we introduce a notion of Contextual Ancestry (CA). Given a concept A from a diagnostic report (e.g. a diagnostic test, a molecular entity, a specific cell), a Contextual Ancestry represents diagnostic concepts that can serve as a context for the concept A. Such contextual concepts are arranged in the CA in the order of their appearance in the reports. For instance, CD4 immunohistochemical (IHC) antibody test can have the following CA:
CD4->Block 1A->IHC study->Lymph Node->Surgical Report Diagnostic concepts Block 1A, IHC study, Lymph Node, Surgical Report can all serve as a context for CD4 IHC test. We have to note here, that Contextual Ancestries are specific to a LIS platform used in a pathology practice, in the sense that they reflect the style and order of sections of a diagnostic report generated by that LIS. Contextual Ancestries can form hierarchical structures like trees or even networks. The above example represent a linear path in the CA hierarchy.
The generated CAs are then incorporated in real time into a Diagnostic Practice Ontology (DPO), which models various types of relationships required to structurize diagnostic reports. 2.2
Structurization of implicit diagnostic information is performed through the Context Resolution step of the pipeline in an automatic fashion (bottom panel in Figure 3). First, a diagnostic report is processed by a structurization pipeline to generate relational triples. To do this, we utilize Stanford OpenIE software. The resulting RDF triples are then arranged as n-ary relation models. Such n-ary models represent structurized version of specific informational points from the diagnostic report. However, while they may successfully convey the intended semantics, in many cases, they lack the contextual component. The lack of contextual information may undermine the usefulness of the n-ary models. To demonstrate this, consider the following excerpt from a diagnostic pathology report (Figure 4):
The report includes a statement about the presence of Reed-Stenberg (RS) cells. The corresponding structurized n-ary model (Microscopic_Description_002) that conveys this information is shown at the right bottom of the upper panel in Figure 5. It can be clearly seen that the structurized version of the report does not include information in which type of tissue RS cells were spotted. This type of critical information is implicitly conveyed by the first sentence in the excerpt (underlined in Figure 4) and noted by a human reader. Other examples include implicit contextual relationships between IHC_Study_001 and Surgical_Report_001, IHC_Study_002 and Surgical_Report_001, Microscopic_Description_001 and Surgical_Report_001, and Microscopic_Description_002 and Surgical_Report_001, which connect specific microscopic findings and test results to patient information. The implicitly reported location of RS cells after context resolution step is properly established as being in lymphoid tissue. The generated contextual relationships are represented by a RDF predicate in_context_of and marked with purple in Figure 5.
The process of generation of contextual links is depicted by a pseudo-code in Figure 6, which represents an (unoptimized) iterative process of finding context entities for each node in the structurized version of a diagnostic report. We call the overall output of the developed pipeline a Knowledge Graph, where all individual structurized informational points are connected in a cohesive manner to holistically represent a diagnostic pathology report.
We analyzed 34 pathology reports that yielded over 3,500 RDF-like relational triples
that we represented as KGs. We have performed an expert assessment of the
effectiveness of conveying implicit contextual information into the generated knowledge graphs
of diagnostic reports. For that, we recruited 6 domain experts from the University of
Missouri Department of Pathology to evaluate the output of our method. The evaluation
was based on three levels of a Likert-like scale: “in context”, “not in context”, “not
clear”. We measured the effectiveness of our model with performance statistics for
information retrieval systems. The precision metrics of matching the semantical content
of the free-text with the corresponding knowledge graphs was 0.92 and the recall was
0.84. A Fisher’s exact test was used to assess statistical significance. The Fisher’s exact
test has odds ratio 19.7 and p-value of 2.2e-16. Inter-Raters’ Reliability (IRR) score
according to a two-way random effects model based on a fully crossed design as
described in [
We have developed a method to model contextual information in order to recover implicit relationships among structurized diagnostic entities. The method enables structurization pipelines to convey contextual information and connect structurized informational point into a cohesive and holistic representation of free-text diagnostic reports, which we call Knowledge Graphs. A limitation of our study is the sample size. Our future efforts will concentrate on applying rules to our ontology. Preliminarily, KGs are important in healthcare for data mining aspects and knowledge acquisition.