Lung cancer is a leading cause of death in developed countries. This paper presents a text mining system using Support Vector Machines for detecting lung cancer admissions. Performance of the system using different clinical data sources is evaluated. We use radiology reports as an initial data source and add other sources, such as pathology reports, patient demographic information and hospital admission information. Results show that mining over linked data sources significantly improves classification performance with a maximum F-Score improvement of 0.057.
Text and data mining are proving to be increasingly important and powerful techniques
for extracting information and insights from Health and Hospital Information Systems
[
Most previous clinical text mining applications have made use of a single textual
data source, e.g., radiology reports, in order to identify or mine information related to a
single condition (e.g., [
In this paper, we describe performance of text mining in the context of the challenge of identifying patients admitted to a hospital for treatment for lung cancer. Lung cancer is a leading cause of death in developed countries, and automatically mapping patient admissions to ICD (International Code of Diseases) directly from hospital records is a precursor to automated ICD-coding, a massively time-consuming manual process at the core of the procedure followed to fund hospitals.
The focus of this paper is to evaluate the value of data linkage and investigate the source of value within different hospital data sources. In particular, we consider a large collection of radiology and pathology reports, along with associated metadata sources, and build classifiers for each type of data source, as well as their combination. Our results confirm that, as might be expected, jointly mining multiple linked data sources improves text classification performance. Analysis also identifies which information source is most valuable for mining for the specified disease, although we expect this to vary with different diseases.
A substantial amount of relevant disease information exists in various types of medical records. Much of this information is in the form of free text; hence text mining represents a promising strategy for building machine learning classifiers that take advantage of the richness of such records. Both radiology and pathology reports have been studied as a source of specific clinical information in previous text mining studies. A pathology report describes the results of examining cells and tissues under a microscope after a biopsy or surgery. A radiology report represents a specialist’s interpretation of images related to a patient’s signs and symptoms.
Hripscak et al. [
In previous work [
The data for this study was extracted from the Alfred Health Informatics Platform,
called REASON [
For the purpose of this study, we extracted textual form of radiology and pathology reports for the financial years 2012-2013 and 2013-2014. Each report was assigned an admission identifier, which is in turn linked to patient metadata. The following metadata associated with each admission were extracted: patient’s demographic data (gender, age, ethnic origin, country, language, marital status, religion, and death date) and hospital-related admission data (hospital code, admission date and time, discharge date and time, length of stay, reason for the admission, admission unit, discharge unit, admission type, source, destination and criteria). Radiology reports were also associated with radiology questions, i.e., a short description of the reason given by the clinician for requesting the scan. The initial number of admission records used in this study was as follows: 40,800 radiology reports; 20,872 pathology reports; and 121,700 metadata entries.
Each admission is associated with a set of ICD-10 codes, which are annotated in the admission record by an internal clinical coder for reporting purposes. These are used in our study as ground truth to build the gold standard data set. The ICD codes are ignored when testing the classifiers – i.e., the classification task consists of identifying those records which contain the ICD code of interest in the gold standard data set.
To identify positive lung cancer cases we used the ICD-10 code C34.*: Malignant
neoplasm of bronchus and lung. In our dataset, only 496 out of 40,800 admissions with
radiology reports were positive for lung cancer. The highly skewed nature of the data
poses a specific challenge to automated machine learning approaches, which generally
perform better over balanced class distributions. To address this problem, we
performed subsampling, randomly selecting a subset of negative admissions to balance
the datasets. Other, more time complex methods (such as oversampling [
Machine learning algorithms require a representation of relevant features of each data point that can be used to build a predictive classifier. The feature representation we adopted for our task combines characteristics obtained from text reports, along with the patient and hospital metadata linked to each admission.
Text in radiology reports, radiology questions and pathology reports was
processed with the MetaMap tool [
Meta Mapping (701):
748 C0559956: Replaced (Replacement) [Functional Concept]
748 C0205090: Right [Spatial Concept]
778 C2316681: Frontal approach [Functional Concept]
We employed the NegEx module to identify the polarity (negative or positive, e.g.,
“Non contrast in the brain”) of phrases. NegEx is a simple algorithm included in
MetaMap that implements several regular expressions that indicate negation, filters out
sentences containing phrases that falsely appear to be negation phrases, and limits the
scope of the negation phrases [
We collected phrases mapped into UMLS concepts for each sentence. Identified phrases were marked with whether the concepts were found in a positive or negative context. Phrases from different reports of the same kind (e.g., radiology reports) belonging to the same admission were merged such that repeating phrase was counted only once. We then built series of feature vectors r[_q][_p][_m]. The r feature vector represents our baseline and contains a “bag” (i.e., an unordered list) of biomedical phrases from radiology reports only. Other feature vectors add the following optional sources: q – radiology questions, p – pathology reports, and m – metadata.
We treated ICD-codes as targets for classification. To identify those data sources that
contain the most valuable information for identifying lung cancer admissions, a
classification framework was built for each feature vector described above.
We used the Weka Toolkit [
Evaluation of TM and NLP systems typically involves the following three metrics: precision, recall and F-Score. Precision of positive/negative class (also called positive/negative predictive value) is the ratio of correctly classified positive/negative values to the number of all instances classified as positive/negative. Recall of positive/negative class is computed as the number of correctly classified instances from the positive/negative class divided by the number of all instances from the positive/negative class; this is also known as sensitivity. F-score is the weighted harmonic mean of precision and recall.
We performed 10-fold cross-validation, where we randomly split data into
train/test halves 10 times. We measured precision, recall and F-Score for each fold. We
calculated statistical significance for F-Score using the Wilcoxon signed-rank test, as
recommended in [
As can be seen, the enhanced classifier performed significantly better than the baseline system (r), with an F-Score difference of +0.023. Similarly, the top right cell shows that a classifier which uses all the features (r_q_p_m) performs better than the classifier without radiology question phrases (r_p_m); however, this difference was not statistically significant.
Our baseline classifier for automatically identifying cases of lung cancer built on only
radiology report phrases shows comparable performance to that in our previous work
[
Not unexpectedly, our results indicate that more informed systems can be built by including multiple data sources. Radiology questions and metadata seem to contain crucial information for detecting lung cancer cases, significantly improving performance when added to radiology reports or to the combination of radiology and pathology reports. The reason for lack of statistical significance when adding pathology reports to train the system may be due to a dearth of pathology reports (only 518 of 992 admissions with a radiology report had pathology reports associated with them).
We have shown that mining multiple linked data sources improves classification
performance of lung cancer ICD-10 codes from textual data, as compared to using a
single data source. We expect similar results for other diseases and plan to use different
ICD-10 codes as targets for classification in our future work. In addition, we plan to
use other techniques to address the problem of highly skewed data sets such as
oversampling [