We present a research prototype for systematic template lling based on in-memory database technology. Entity extraction and normalization is based on domain-speci c dictionaries and customized rules set building on top of related work of the medical eld. The prototype called HPI proves feasibility of in-memory technology to enhance work ows in the eld of e cient text processing and analysis. With our approach, the iterative process of dictionary and rule re nement for enhancing text analysis results shifts from a time-consuming task with long waiting hours to a continuous work ow. In the context of the challenge's task, our prototype achieves an overall average accuracy of 0.769 and an overall F1 measure of up to 0.323.
Professional health care requires a constant documentation of all patient-related data, such as history of clinical events. This clinical data is stored in a humanreadable format, such as text les, since it supports the daily work of the clinical personnel. This data is only available in an unstructured format, which makes its automatic processing a complex task. However, for the sake of fault prevention, comparison, performance optimization, and subsequent clinical research, the important information must be e ciently extracted from the unstructured data for further processing. This task requires methods from Information Extraction (IE), which is a speci c subdomain of Natural Language Processing (NLP).
The second task of the 2014 CLEF eHealth challenge requires the extraction
of information from unstructured clinical data to ll speci c templates, i.e. xed
sets of di erent semantic classes depending on the IE purpose [
We as team HPI participated in the context of a student internship in
this challenge. We designed a research system incorporating lasted In-Memory
Database (IMDB) technology to enable systematic lling of templates of the
required classes using unstructured data from Electronic Medical Records (EMR).
IMDB technology has proven to have major advances for analyzing big
enterprise and medical data, e.g. to support medical doctors in identifying better
treatments for cancer patients and other elds of life sciences [
In the following, we describe data used in our system, its architectural details, and highlight the advantages of using IMDB technology. 2.1
For the training phase, we used a data set of 300 documents taken from
version 2.5 of the Multiparameter Intelligent Monitoring in Intensive Care (MIMIC
II) database [
In addition to the training data, we integrated the SNOMED Clinical Terms
(SNOMED CT) data from the Uni ed Medical Language System (UMLS)
version 2013AB to improve our entity recognition capabilities for mentions of
diseases and body locations [
Lightweight Compression Those techniques refer to a data storage
representation that consumes less space than its original pendant. A columnar database
storage layout supports such lightweight compression techniques, e.g. dictionary
encoding which maps all unique values to a uniform format [
Multi-Core and Parallelization Modern system architectures are designed to provide multiple CPUs with each of them having separate cores. This capacity should be fully exploited by parallelizing application execution to achieve maximum processing speed. The incorporated IMDB platform supports this and provides built-in parallelization. With that, we do not need to apply parallelization strategies on our own but still have maximum runtime performance in processing our input data.
Entity and Feature Extraction Any kinds of text, such as the medical reports that have to be processed in this challenge, are considered as unstructured data. Thus, it cannot be processed automatically unless a machine-readable data model exists for automatic interpretation, e.g. a semantic ontology. Our incorporated IMDB platform o ers a range of features for text processing, of which the relevant ones for us are those for entity and feature extraction. Entity and feature extraction refers to the identi cation of relevant keywords and names of entities from documents. Dictionaries and individual extraction rules can customize this. Dictionaries list one or more entity types, each of which containing any number of entities that in turn contain a standard form name and any number of synonyms. Extraction rules use formal syntax to de ne entities of a speci c type. This allows formulating patterns that match tokens by using a literal string, a regular expression, a word stem, or a word's part of speech. 2.3
Fig. 1 presents our system architecture in Functional Modeling Concepts (FMC)
notation [
In-Memory Database Relevant data is imported into our IMDB. The data is comprised of the medical reports whose templates must be lled, the SNOMED CT subset, and a list for each slot type with entities that have been extracted from the training data before. From this data, we create scienti c medical dictionaries and add individual extraction rules to facilitate entity recognition and extraction of the di erent slot types.
Dictionaries We build customized dictionaries to identify slot types NI, SC,
UI, CC, SV, CO, GC, BL, and TE in the given medical reports. For extraction
and normalization of DD and BL slot types, we compile a dictionary based
on the imported SNOMED CT data set. Entities of remaining slot types are
extracted and normalized by a dictionary derived from training data. Fig. 2a
depicts such a dictionary in XML format. Entities can easily be organized into
categories, normalized by a standard form and enriched by additional variant
de nitions. The given example lists the semantic type Body Part, Organ, or
Organ Component in blue letters. Afterwards, the concept de nition with its
normalization, i.e., the standard form, is de ned in black letters. Finally, possible
entities owning the de ned normalization are listed in yellow letters. As a result,
the phrase skeletal muscle structure of abdomen has the normalization C0000739
and will be assigned to the BL slot type when detected in any text document.
CGUL Rules We de ne extraction rules in Custom Grouper User Language
(CGUL) to identify DT slot types [
Entity Recognition and Extraction With the created dictionaries and CGUL rules
at hand, we can trigger the actual process of entity extraction within our IMDB.
For that, we create a full text index on the medical reports for which we have to
ll out the templates [
Template Filling Our template lling engine is based on Python v. 2.7.7 and takes extracted entities together with their normalization, slot type, medical report it occurred in, and location within the medical report, i.e. text spans, as input. These details are associated to the corresponding templates, which requires matching the text spans identi ed by our approach. The DD mentions provided by default in the test data are used as "anchor" to determine entities of the same template. If this has been accomplished, the templates are lled with the corresponding cues and normalizations. 3
In the following, we present experiments conducted in terms of data and evaluation metrics used. We provide experiment results according to the presented metrics and discuss relevant ndings. 3.1
For evaluating the performance of our system, we used a test data set provided by the challenge. Analogously to the initial training data, this data set is comprised of a set of 133 medical reports with template documents assigned. In contrast to the training data, the templates' attributes, i.e. cue and normalization values for each slot type, are empty. In our experiments, we aim at lling both cue and normalization values and by that to participate in tasks 2a and b of the challenge.
We use accuracy and F1 measure as common measures used in pattern
recognition and information retrieval for evaluation of the derived normalization and
cue values, respectively [
Accuracy(N ) = jNcorrectj jN j (1)
Eq. 2, Eq. 3, and Eq. 4 depict computation of F1 measure to assess quality of the detected cue values, which is the harmonic mean of precision and recall. With regards to examining performance for a concrete slot type or their overall set, C is the set of all cue values detected by our approach, whereas Ctrue contains all true cue values. Ccorrect is the set of all cue values that have been correctly identi ed by our approach and is also expressed as Ccorrect = Ctrue \ C. The de nition of the term "correct" varies for strict and relaxed evaluation. The former checks if a derived cue value equals the correct one, whereas the latter still considers a cue value as correct if it overlaps with the true value. Precision depicts the fraction of retrieved instances that are relevant, i.e., in this context how many of the true cue values have been identi ed by our approach. Recall depicts the fraction of relevant instances that are retrieved, i.e. how many of the cue values identi ed by our approach are contained in the set of "true" cue values.
F1(C) = 2
Recall(C) P recision(C)
Recall(C) + P recision(C)
Recall(C) = P recision(C) =
jCcorrectj jCcorrectj + jCtrue n Cj
jCcorrectj jCcorrectj + jC n Ctruej (2) (3) (4) 3.2
The results achieved by our prototype are summarized in Tab. 3.2 and Tab. 3.2 for tasks 2a and 2b, respectively. For many of the slots, our results for task 2b, i.e. cue values derived, were quite lower than the ones obtained for task 2a, i.e. normalized values derived. For instance, we achieved 90-100 percent of accuracy for the slot types CC and GC, but only 21 and 14 percent F1-measure, respectively, for the relaxed evaluation of task 2b. Although this is expected, as exact (or relaxed) mention spans are harder to be correctly extracted than the corresponding normalized values, we still investigate possible mistakes on the o sets in our submissions and future error analysis will shed some light on the discrepancies between the results for both tasks.
Our strategy for the BL slot, which had relied on the dictionaries derived from the SNOMED CT terminology, achieved 50 percent of accuracy. A future error analysis will also show whether false negatives were due to concepts that are not present in the SNOMED CT terminology, to missing synonyms for existing concepts or on the matching approach that was used. Nevertheless, the relaxed evaluation of task 2b shows that our dictionary matching approach provides good precision, i.e. 60 percent, given the complexity of the anatomical nomenclature.
Extraction of values for slot type DT was a hard task and results were quite low for all teams. This is because it requires a more careful analysis of the language, such as analyzing verb tenses and time expressions. However, we believe that our approach of using CGUL rules is appropriate for extracting this information but more rules should be created for this purposes as well as a revision of the existing ones.
Therefore, the used dictionaries and rules instead of the underlying IMDB system induce the presented results. If those dictionaries are re ned, e.g. by including other data sources than SNOMED CT or adapting extraction rules, we are convinced that the overall performance of our system will improve.
However, the focus of this work is rather on showing the general applicability and feasibility of in-memory technology for processes that involve processing and analysis of unstructured text. One iteration to improve text analysis results, starting with re ning dictionaries and ending with receiving the nal results, i.e. the lled templates from the test data, takes minutes with our system instead of hours or days with traditional approaches. This proves that in-memory technology provides advantages also for the eld of information extraction and can contribute to establishing e cient and alternative processing strategies in that area. 4
The ShARe/CLEF eHealth challenge 2014 aims to facilitate the research on information extraction within the biomedical domain. As follow-up to 2013's challenge, participants were asked to identify semantically related mentions to disorder mentions and ll out templates with normalization and cue values for the detected entities.
In the context of a student internship, we designed a research prototype for entity extraction based on IMDB technology that proves feasibility for e cient text processing. Evaluation results show that our rules and dictionaries currently applied require optimization by re ning dictionaries or extraction rules. However, our prototype allows us to extend existing extraction rules and dictionaries in a constant manner and to verify them instantly. Thus, the task of iterative improvement of text analysis results becomes a continuous process.