The current clinical practice requires physicians to gather, interpret and correlate information from multiple independent multimedia data sources to manage patients. Due to poor structuring and organization, it is too timeconsuming to access the information snippets embedded in the various pieces of data in the longitudinal patient records. This becomes more of a problem when correlating the temporal progression of various factors obtained from patients clinical, laboratory, imaging and genomics studies. Making such correlations is an essential component of the prognosis and treatment planning tasks in patient care. In addition, the similarities in the disease progression pattern among different patients and their relationships to outcomes remain hidden from the clinicians in the piecemeal use of the data. We believe that there is a gap between the decision-enabling information and insight required for efficient patient management and the heterogeneous data comprising the patient records that can be bridged with advanced multimodal content analytics, semantic information organization, summarization, and visualization tools. In this paper we present a case study in organizing, accessing, and visualizing information obtained through structuring the multimedia and multimodal data for brain tumor patient management and how such information map to the needs of the clinicians. We report our early work on the analytics, user interface and the preliminary evaluation results which indicate that the presented approach caters well to the clinician needs for the task of brain tumor patient management.
The current clinical practice of neuro-oncology requires physicians to correlate and interpret information from multiple independent data sources to diagnose, treat and manage patients with brain tumors. The heterogeneous sources of data needed for these complex tasks are multi-media and multi-modal 1 in nature. Each piece of data intends to capture specific information about the health status of the patient (primary information), or reflect the decisions made by the clinician at a given point in time (secondary information). These information when properly linked, correlated and summarized can form an overall picture of the patient’s condition and provide insight on the effectiveness of the course of treatment or the natural progression of the disease, which can then be used as the basis for making further clinical decisions.
Today, access to these information resources is typically a manual process;
regardless of how technologically advanced a health-care center is in adopting the
Electronic Patient Record (EPR) [
The purpose of the longitudinal medical records is to document the various aspects of
the patient’s condition over time. The patient condition and overall status is assessed
from the collection of information snippets embedded in the various pieces of data at
any given time. For every given sub-speciality, there are certain concepts of interest
along with their attributes that are being used to assess the patient condition. There are
many existing ontologies used today to represent concepts relevant to human anatomy,
disease, treatment plans and procedures, such as FMA [
In this work, we aim at representing and linking instances of concepts of interest
and their attributes as manifested in the heterogeneous data sources. We define the
notion of a ConceptFrame that affords knowledge-guided extraction of medical concepts
(along with their associated attributes) from heterogeneous medical artifacts. Figure 2
illustrates the idea of the ConceptFrame for the concept edema. As shown in this
figure, there is a mention (instance) of edema in the MRI of type FLAIR, which visually
captures the region corresponding to the concept of edema [
The task of instantiating ConceptFrames and populating them is peformed by
analytic engines that specialize in finding instances of the concepts in various sources of
multimedia/multimodal data. For the type of data we have in this study, i.e. MR
images and oncology notes, we need image analytics that can find instance of the concept
edema in the volumetric images of the brain and text analytics that find concepts such
as drug, diagnosis, etc. Figure 3 shows the architecture of the COSMus system, which
is responsible for applying the appropriate analytics to the right data source and
instantiating the ConceptFrames. This architecture illustrates our final goal. In the current
work we have a limited proof-of-concept implementation to let us study the effect of
organized and linked information in the clinical practice. In the following section we
present our work on using transductive and inductive mechanisms for concept
identification in multi-protocol MR images. For text analytics, we refer the reader to our prior
work in medical text analytics [
A number of methods to detect and track changes in MR images have been proposed [
Recently, semi-supervised learning methods such as transductive inference have
been getting a significant amount of attention given their effectiveness on quickly
labeling a given set of input data. A transductive method minimize the human interaction by
inferring the labels of a complete dataset from a small initial expert input [
For locating and characterizing the concepts of interest which are manifested in
longitudinal MR images we combine the transductive and inductive learning techniques.
After the registration phase, which aligns the multi-modal set of images at a given
timepoint into a common coordinate system, we capture minimal input provided by domain
experts for identifying the concept of interest (in this case the edema region). We then
use a Bayesian transductive learning approach [
The transductive approach for obtaining the pseudo ground truth shows promising
results with a sensitivity and specificity of 90.37% and 99.74% respectively (see [
Leveraging the pseudo-ground truth resulting from using the transductive method on a few images and creating inductive (SVM) classifiers for the concept of edema, produced a classifier which was able to classify other images in the longitudinal patient records with a minimum accuracy of 80%, when the SVM classifier was trained using pseudo-ground truth for 3 time points. This indicates that one can capture limited amount of expert annotations and train a classifier that can generalize well to other images, without overwhelming the domain experts. 4
We have built and tested a prototype system for analyzing and displaying patient images and text-based oncology reports of brain tumor patients accumulated over time. The text analysis includes the extraction of neurological symptoms, drug and dosage information, lab results, vital measurements and assessments. This is coupled with the ability to scroll through 3D MRI images, plot and highlight edema volumes and plot any recurring numerical value, such as weight, platelet count and blood pressure. Limited clinical evaluation has been performed to validate the usability of the prototype system in clinical decision making. During the initial presentation to members of the team, several remarked that the ability to plot data such as edema volume and drug dosage led to immediate and unexpected insights. Both subjects were able to use the system effectively with minimal training. The experimenters provided assistance only when the subjects were stumped. The complete session with each subject was recorded, and notes were taken by the investigators. Although the participants were able to employ most system functions, they experienced a range of problems. The semantic mappings used to label buttons that subsume findings were not always intuitive. For example, both subjects expected that the ”seizures” findings would be subsumed under the Neurology tab when in fact they were listed under ”Symptoms” as it was in the oncology notes. Both subjects had some difficulty discerning the date of each time point they were examining as that date was displayed just under the accordion control rather than closer to the actual time data point. If they were placed in such proximity, they would have overlapped each other.
Initally, the neuro-oncology resident experienced some difficulty selecting and juxtaposing images in the 2 panels. However after a short period of time, she had no difficulty finding the edema information and was able without any assistance to make a comparative plot of edema volume and Avastin dosage. She was also able draw appropriate inferences about changes in the patient’s condition over time.
In summarizing the patient’s status in question 7, she readily grasped that the current status would be available by clicking on the last data point. She noted that the best part of the interface was the way it integrated views of text and image data, and that it was most useful to be able to look at any pairs of images together. She remarked ”Being able to determine your thrombocytopenia at that point and the edema volume feature is the best aspect of this whole thing.”
The radiology fellow was more interested in examining the images. He showed less interest in the symptoms reported as text in the accordion tabs. He readily selected among the FLAIR, T1C+ and T1C- images and found them easy to study. In general, it took him more time and mouse clicks to find what he was looking for and longer to develop a basic mastery of the system. The system works by placing the image from the first thumbnail you select in the left image box and the second on the right. Then if you select a third image, it replaces the first one. The system provided insufficient feedback and guidance and he found it a source of confusion. He also noted that the ”Lock” checkbox which decouples one image from the other worked exactly the opposite of the ”Link” checkbox provided in the GE PACS system. This subject was more conservative in drawing inferences from the images. In this system, the total edema volume was computed semi-automatically as described above. He commented ”everything that this machine is calling edema is not necessarily relevant to the patient’s survival because that could just be the radiation if that’s in the radiation field. The part of FLAIR signal abnormality which is in the field of the tumor or on the tumor may be more relevant, but once it becomes confluent, you can’t tell which one is tumor and which one is radiation change.” He scrutinized the images more carefully and commented on the resolution and potential noise that made clinical inferences less certain. He was also less sure about the value of integrating the text and image modalities because his work was almost entirely with the images. Despite his critical comments, he remarked that the system offered great potential as a clinical tool. His comments offered many excellent insights into the improvement of the interface.
On the Likert survey, both subjects rated the system very highly in terms of ease of use, learnability and were especially appreciative of the ability to integrate disparate sources of data. They were less convinced that this would be an effective tool to discover novel dimensions of the patients’ problems. Both residents agreed emphatically that this was a tool with great clinical potential.
In this paper we presented a set of ideas and preliminary results in organizing, summarizing, and visualizing longitudinal multimedia data sets in the context of brain tumor patient management. The initial results of our work on extracting information from multimedia and multimodal medical data and organizing them around a set of concepts of interest shows promise in the value of such approaches for aiding the clinicians in understanding the patient data better, which evenntually result in making better clinical decisions. We demonstrated that one can acheive a balance with obtaining limited annotations from the domain experts on the concepts of interest and leveraging them to desing a classifier for concept identification in patient’s unseen longitudinal data sets. In addition the display and communication of the concepts and their attributes, rather than raw data, was helpful in expediting the time to information and aiding the clinicians see co-evolutions of clinical factors derived from the data. We plan to fully implement the COSMus system along with the supporting analytics for the domain of brain tumor data management.