=Paper=
{{Paper
|id=Vol-2399/paper07
|storemode=property
|title=Biomedical Data Categorization and Integration using Human-in-the-loop Approach
|pdfUrl=https://ceur-ws.org/Vol-2399/paper07.pdf
|volume=Vol-2399
|authors=Priya Deshpande
|dblpUrl=https://dblp.org/rec/conf/vldb/Deshpande19
}}
==Biomedical Data Categorization and Integration using Human-in-the-loop Approach==
https://ceur-ws.org/Vol-2399/paper07.pdf
ceur-ws.org/Vol-2399/paper07.pdf
Biomedical Data Categorization and Integration using
Human-in-the-loop Approach
Priya Deshpande
Supervised by Dr. Alexander Rasin
DePaul University
Chicago, IL, USA
pdeshpa1@depaul.edu
ABSTRACT However, in the healthcare domain, datasets are often not shared
Digitized world demands data integration systems that combine because of security concerns, lack of integration, or limitations of
data repositories from multiple data sources. Vast amounts of exist- retrieval engines. A data integration framework should make data
ing clinical and biomedical research data are considered a primary available, accessible, and support fine-grained access control for
force enabling data-driven research toward advancing health re- different users [6]. It would also greatly reduce the need for man-
search and for introducing efficiencies in healthcare delivery. Data- ual curation of data sources and data repositories. Data integra-
driven research may have many goals, including but not limited to tion alone is insufficient without associated information retrieval
improved diagnostics processes, novel biomedical discoveries, epi- mechanisms that would rank retrieved results based on relevancy.
demiology, and education. However, finding and gaining access to From our discussions with University of Chicago (UofC) radiol-
relevant data remains an elusive goal. We identified different data ogists, even the internal UofC commercial system lacks some of
integration challenges and developed an Integrated Radiology Im- the Natural Language Processing (NLP) features (e.g., detecting
age Search (IRIS) framework that could be a step toward aiding synonyms and negation) and multimodal (text and image) search
data-driven research. We propose building a biomedical data cate- capabilities. We studied publicly available radiology data sources
gorization and integration framework using human-in-the-loop and MyPacs.net2 , EURORAD3 , and RSNA Medical Imaging Resource
developing data bridges to support search and retrieval of relevant Community (MIRC)4 , that provide a collection of clinical reports
documents from the integrated repository. and associated images, which are known as teaching files. Teaching
My research focuses on biomedical data integration, indexing files contain information such as patient history, findings, diagno-
systems, and providing relevance-ranked document retrieval from sis, differential diagnosis, or discussion notes. While all of these
an integrated repository. Although we currently focus on integrat- public data sources are available, most of them provide only basic
ing biomedical data sources (for medical professionals), we believe search capabilities – not offering NLP support or ranked retrieval
that our proposed framework and methodologies can be used in mechanisms. Several studies highlighted the need to integrate clin-
other domains as well. ical reports and images into databases with advanced search ca-
pabilities. Gutmark et al. [5] argued for building a system that
PVLDB Reference Format: reduces errors in radiological images interpretation using teaching
Priya Deshpande. Biomedical Data Categorization and Integration using file databases. Talanow et al. [12] described reference radiological
Human-in-the-loop Approach. PVLDB, 12(xxx): xxxx-yyyy, 2019.
DOI: https://doi.org/10.14778/xxxxxxx.xxxxxxx
image use for diagnosis, teaching needs, research, and the resulting
need for an advanced reference search engine.
1. INTRODUCTION An integrated repository of teaching files can retrieve thousands
A growing amount of available biomedical data poses new chal- of results for a text search. A search can thus become effectively
lenges in data management. Data re-usability is a highly desir- useless without being able to show the most relevant results first.
able goal, both for advancing science as well as for replicating Publicly available radiology teaching file search engines do not
or validating results of previous studies. Recognizing this need, provide text relevance ranking or combined text-and-image search.
publishers and funding bodies may require researchers to submit Lack of such systems motivated us to build Integrated Radiology
data generated in their work and make it available to the research Image Search (IRIS) and develop the ranking algorithm presented
community. For example, National Institutes of Health (NIH) is here. We presented IRIS at the annual Society for Imaging Infor-
encouraging funded investigators to use cloud computing to con- matics in Medicine (SIIM 2018) meeting (two posters: one focus-
duct research and make their work accessible to larger audiences1 . ing on search and another on data integration) and received feed-
1 https://commonfund.nih.gov/strides/
back from doctors indicating that this work would be useful for the
medical domain practitioners.
This work is licensed under the Creative Commons Attribution-
NonCommercial-NoDerivatives 4.0 International License. To view a copy
of this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. For 2. BACKGROUND AND RELATED WORK
any use beyond those covered by this license, obtain permission by email- In this section we discuss papers that addressed the need for data
ing info@vldb.org. Publication rights licensed to the VLDB Endowment. integration and retrieval systems along with an overview of exist-
Proceedings of the VLDB 2019 PhD Workshop, August 26th, 2019. Los ing medical data retrieval systems. Several studies have highlighted
Angeles, California. Copyright (C) 2019 for this paper by its authors.
Copying permitted for private and academic purposes 2 https://www.mypacs.net/
Proceedings of the VLDB Endowment, Vol. 12, No. xxx 3 https://www.myesr.org/eurorad
ISSN 2150-8097. 4 http://mirc.rsna.org/query
DOI: https://doi.org/10.14778/xxxxxxx.xxxxxxx
.
the need for integration of healthcare data [10]. Holzinger et al. [7]
talked about knowledge discovery and interactive data mining tech- ID Summary Table 1: Research work summary
niques in bio-informatics, the challenges to integrating biomedical IRIS 1.0
data, and open research directions. Li et al. [8] proposed a hybrid Teaching file text pre-processing and indexing.
human-machine data integration approach that integrates records 1 Smart search through substitution of synonyms
from databases with similar data types (e.g., iphone users data). and interpreting negation. Query expansion using
However, healthcare domain data integration needs to combine het- RadLex through an exact term match. [1]
erogeneous data sources with different categories of data types. IRIS 1.1
Simpson et al. [11] proposed a multimodal image retrieval system Query synonym expansion. SNOMED CT ontology
that retrieves biomedical articles used in Open-i5 . Ling et al. [9] 2
integration, shown improved results compared with
designed GEMINI, an integrative healthcare analytics system, and other search engines [3].
studied problems related to healthcare data heterogeneity and data Data integration as an iterative process, showing how
integration in that context. From this literature survey, we con- 3
each integration step improved IRIS results [2].
cluded that healthcare needs are not met by the current search en- Cluster analysis and coverage analysis for
gines. The limitations of existing systems motivated us to design both ontologies and radiology data sources.
and develop a radiology multimodal search engine. IRIS integrates 4 Unsupervised machine learning to identify data source
two well-known public data sources MIRC and MyPacs and two properties – to identify best data sources and ontologies
medical ontologies RadLex6 and The Systematized Nomenclature for integration (Journal paper – under review).
of Medicine Clinical Terms (SNOMED CT)7 . RSNA MIRC: Pub- IRIS 1.2
licly available large repository with more than 2,500 teaching files Multimodal ranked retrieval for integrated
and more than 12,000 images. 5 radiology data sources using context of search term by
Mypacs.net: Publicly available teaching file resource with more considering weighted ontology and category terms
than 35,000 cases and 200,000 images. (Conference paper – under review).
RadLex: RadLex is an ontological system that provides a compre- Toward using FAIR Principles for Fine-Grained
hensive lexicon vocabulary for radiologists. 6
Access to aid Biomedical Data Driven Research [4].
SNOMED CT: ontology provides a standardized, multilingual vo-
cabulary of clinical terminology that is used by physicians and
other healthcare providers for the electronic exchange of clinical For each phase we have identified a research question. Publications
health information. related to this work are briefly summarized in Table 1
3. METHODOLOGY AND RESEARCH 3.2.1 Design an integrated smart database with het-
erogeneous data sources
STEPS
Research question #1: How to determine which data sources
In this section, we discuss major biomedical data sources and
and ontologies need to be integrated?
significant goals that we identified as a part of my PhD proposal.
Most hospitals maintain a collection of teaching files, but many
3.1 Datasets public teaching file collections are also available through curated
online sources (e.g., RSNA MIRC, MyPacs, and EURORAD). We
We currently focus on three types of data a) Electronic health
developed IRIS engine as a pilot for a data integration system for
records; b) Radiology teaching files or teaching files used by doc-
the healthcare domain [1]. In IRIS, we captured heterogeneous data
tors and radiologists; c) Research datasets.
from MIRC and MyPacs data sources, loading data into an inte-
Electronic Health Records (EHRs): An electronic health record is
grated data repository. Using medical ontologies, we built our own
a digital version of a patient’s record. EHRs are maintained at hos-
dictionary which maps terms to their synonyms from the datasets
pitals and provide patient information such as history of patient,
and medical ontologies [3]. We designed an unsupervised machine
medical test results, allergies, immunization details, radiology im-
learning technique that performs coverage analysis of data sources
ages, and clinical reports.
and medical ontologies to learn properties of the data (e.g., topic
Medical Teaching Files: A radiology teaching files system is a
coverage). By learning data repositories contents, one can decide
collection of important cases for teaching and clinical follow-up.
which data sources need to be integrated or what repository con-
Teaching files share a similar overall structure but significant vari-
tent is lacking. Thus, this coverage analysis algorithm benefits data
ations exist even within the same data sources and can include in-
integration process by extracting knowledge about the repositories
formation such as patient history, findings, diagnosis, discussion,
(addressing research question #1). Our analysis also confirmed that
comments, references, and images related to clinical reports.
data integration is a continuous, iterative process [2].
Research datasets: From our survey with different research institute
datasets, we observed that most of the data in healthcare domain 3.2.2 Ranked retrieval search engine with multimodal
are images (e.g., CT, X-ray, MRI). Those images are most typi- text and image-based search capabilities
cally stored in formats such as JPEG, DICOM, or PNG and include
associated text data describing patient and case information. Research question #2: How to find relevant documents given a
keyword query or hybrid (text+image) query? Figure 1 shows the
3.2 Data integration and rank retrieval architecture of IRIS engine. When a user enters a text query, IRIS
We have organized this project into three phases (I finished the performs query expansion using relevant ontologies, and retrieves
first two phases and working on the last phase of my PhD work). relevant results to the query term. Our database also stores accuracy
feedback from users which is then used to evaluate and iteratively
5 https://openi.nlm.nih.gov/
improve IRIS results.
6 http://www.radlex.org/ An integrated search may result in thousands of matches; thus,
7 https://www.nlm.nih.gov/healthit/snomedct/ we are designing a search algorithm that ranks results by incorpo-
on defining standard data cleaning technique that would be applica-
ble to the most of the similar data sources that we proposed in this
work. Our data categorization module categorizes data items into
different sets based on the usage of those data elements in search
operation. We need support from a human to check the accuracy of
data categorization, to set similarity thresholds between different
data items, and apply additional domain knowledge to categorize
these data items based on relevance between data objects. Our data
categorization algorithm will differentiate data items based on di-
agnostic relevance. For example, teaching cases with title, findings,
and diagnosis would be treated as one sub-category in teaching
cases (that would also integrate clinical reports) while another sub-
category could integrate fields those are medically less relevant e.g.,
Figure 1: IRIS Architecture discussion, history, or comments. Based on data categorization we
will be designing database schema and would also evaluate schema
based on standard database schema benchmark techniques. Data
rating context computed through a weighted ontology terms. For write bridges would be responsible for the extracting data from dif-
text-based search ranking evaluation we used Normalized Discounted ferent data categories and loading data to the respective database
Cumulative Gain (NDCG)8 algorithm to measure the quality of schema. This data categorization work is ongoing and we do not
search result ranking. Our analysis showed an improvement in have any experimental results yet. We will address research ques-
ranked retrieval as compared to other search engines (addressing tion #3 by implementing this module.
research question #2).
3.2.3 Data bridges and indexing mechanism to inte- 4. EXPERIMENTAL RESULTS
grate biomedical data sources In this section we briefly discuss the current results from pro-
Research question #3: How data integration performance (time) posed system.
and scalability (adding variety of data sources) can be improved us-
ing data bridges? In order to make our integration solution applica- 4.0.1 Text-based results
ble to other biomedical data sources (e.g., EHR’s, clinical reports), We evaluated IRIS search ranking using a combination of queries
we plan to create data adapters that will serve as a bridge between received from radiologists at a well-known hospital and other queries
data providers and data integration systems (this work was a part of chosen from an extensive literature survey. We have initially tested
my internship at NIH). Data providers can share their data in any a total of 28 text queries, out of which we picked a subset of 10
file format and bridges will interpret that data in a uniform manner. queries (Q1:Cardiomegaly, Q2: ACL Tear, Q3: Annular Pancreas,
As shown in Figure 2, our data clustering indexing approach starts Q4: Pseudocoxalgia, Q5: Varicocele, Q6: Angiosarcoma, Q7: Tra-
cheal dilation, Q8: Appendicitis, Q9: Bronchus intermedius, Q10:
Cystitis glandularis) to perform an in depth evaluation. Due to
space constraints we briefly discuss text based results. We evalu-
ated text-based results on a scale from 0 (“not relevant”) to 2 (“very
relevant”). We defined five categories to score text search results:
“not relevant” = 0 (when term and synonyms do not appear any-
where in the results), “relevant” = 0.5 (if term or synonyms appear
in any category of teaching file), “more relevant” = 1 (if term or
synonyms appear in discussion category), “most relevant” = 1.5 (if
term or synonyms appears in history or ddx category), and “very
relevant” = 2 (if term or synonyms appears in title, findings, or di-
agnosis categories).
Comparison of IRIS and MIRC relevance rank algorithm us-
ing same datasets:
We compared IRIS relevance rank algorithm with MIRC using the
same dataset. We considered top four teaching file results from
IRIS, MIRC, and Google site search. We calculated relevance score
Figure 2: Data Categorization with human-in-the-loop by scoring top four teaching files from each engine, using weighted
ontology ranking algorithm . Figure 3 shows an overall analysis
with collecting different biomedical data sources. From our liter- of results from these 3 search engines. score for each search en-
ature survey we observed that data preparation accounts 80% of gine shows that IRIS relevance rank algorithm performs better than
data scientist work. Data preparation includes finding relevant data other two engines.
sources, extracting data from those data sources, data cleaning, and Ranking evaluation of other medical search engines:
data integration. Our proposed data integration system would help We also considered how other public medical radiology teach-
data scientists and researchers optimize and streamline data prepa- ing file search engines rank their search results. We used the same
ration. We collected different biomedical data sources and working query set and performed a search using MIRC, MyPacs, EURO-
RAD, and Open-i search engines. We discuss only two queries
8 https://en.wikipedia.org/wiki/Discounted_ (Q1:“cardiomegaly” and Q8:“appedicitus”) in detail and reporting
cumulative_gain scores for the top 10 search results. Figure 4 shows a comparative
weight to ontology terms we show that teaching files can be better
ranked in order of their relevance to a search query. Currently I
am working on data write bridges and categorization algorithm to
improve biomedical data integration process.
6. ACKNOWLEDGMENTS
This research was supported in part by the Intramural Research
Program of the National Institutes of Health (NIH), National Li-
brary of Medicine (NLM), and Lister Hill National Center for Biomed-
ical Communications (LHNCBC).
7. REFERENCES
Figure 3: IRIS relevance rank results comparison with MIRC
[1] P. Deshpande, A. Rasin, E. Brown, J. Furst, D. Raicu,
S. Montner, and S. Armato III. An integrated database and
smart search tool for medical knowledge extraction from
radiology teaching files. In Medical Informatics and
Healthcare, pages 10–18, 2017.
[2] P. Deshpande, A. Rasin, E. Brown, J. Furst, D. S. Raicu,
S. M. Montner, and S. G. Armato. Big data integration case
study for radiology data sources. In 2018 IEEE Life Sciences
Conference (LSC), pages 195–198. IEEE, 2018.
[3] P. Deshpande, A. Rasin, E. T. Brown, J. Furst, S. M.
Montner, S. G. Armato III, and D. S. Raicu. Augmenting
medical decision making with text-based search of teaching
file repositories and medical ontologies: Text-based search of
radiology teaching files. International Journal of Knowledge
Figure 4: Rank retrieval score results from other medical search Discovery in Bioinformatics (IJKDB), 8(2):18–43, 2018.
engines [4] P. Deshpande, A. Rasin, J. Furst, D. Raicu, and S. Antani.
Diis: A biomedical data access framework for aiding data
driven research supporting fair principles. Data, 4(2):54,
analysis of ranked results from these four engines using the rele- 2019.
vance scores based on our metric described above. Open-i can rank [5] R. Gutmark, M. J. Halsted, L. Perry, and G. Gold. Use of
search results based on different categories (e.g., based on diag- computer databases to reduce radiograph reading errors.
nosis or based on teaching file date) – we used a diagnosis based Journal of the American College of Radiology, 4(1):65–68,
search in Open-i. MIRC ranks results based on the date of modifi- 2007.
cation with no other option available. Our analysis shows that none
[6] J. R. Hemler, J. D. Hall, R. A. Cholan, B. F. Crabtree, L. J.
of the search engines return the most relevant results first. Interest-
Damschroder, L. I. Solberg, S. S. Ono, and D. J. Cohen.
ingly, top results are often less relevant than the subsequent search
Practice facilitator strategies for addressing electronic health
results. For example for “cardiomegaly” MyPacs fourth result is
record data challenges for quality improvement:
more relevant than the top three results. EURORAD does not re-
Evidencenow. The Journal of the American Board of Family
trieve any results for “cardiomegaly” but we checked “appendicits”
Medicine, 31(3):398–409, 2018.
results – and those were also not ranked based on the relevance of
the search term. [7] A. Holzinger, M. Dehmer, and I. Jurisica. Knowledge
discovery and interactive data mining in
4.0.2 Hybrid Text and Image based results bioinformatics-state-of-the-art, future challenges and
IRIS hybrid algorithm augments the text search with image search research directions. BMC bioinformatics, 15(6):I1, 2014.
and re-ranks results based on the relevance to the query. Due to [8] G. Li. Human-in-the-loop data integration. Proceedings of
space constraints we briefly discuss hybrid search result. IRIS text- the VLDB Endowment, 10(12):2006–2017, 2017.
based and hybrid search results scored an score of 0.83 out of 1. [9] Z. J. Ling, Q. T. Tran, J. Fan, G. C. Koh, T. Nguyen, C. S.
Image search scored only about 0.53 out of 1, validating our use of Tan, J. W. Yip, and M. Zhang. Gemini: an integrative
the image search as an enhancement to the text search (rather than healthcare analytics system. Proceedings of the VLDB
a standalone search). Hybrid search scored 0.84 out of 1 because of Endowment, 7(13):1766–1771, 2014.
text results were augmented with image-based results. For hybrid [10] I. Merelli, H. Pérez-Sánchez, S. Gesing, and D. DAgostino.
search Some of the results were noticeable better than text-based Managing, analysing, and integrating big data in medical
search.By combining text search with image results, we are striv- bioinformatics: open problems and future perspectives.
ing to get a text-based match that also includes a similar image. BioMed research international, 2014, 2014.
[11] M. S. Simpson, D. Demner-Fushman, S. K. Antani, and
5. CONCLUSIONS G. R. Thoma. Multimodal biomedical image indexing and
retrieval using descriptive text and global feature mapping.
The ranking approach presented in this paper is significant be- Information retrieval, 17(3):229–264, 2014.
cause it enables IRIS to present the user with top relevant refer- [12] R. Talanow. Radiology teacher: a free, internet-based
ence cases first. Through integrating term frequency, adding more radiology teaching file server. JACR, 6(12):871–875, 2009.