Proper evaluation is crucial for developing high-quality computerized text summarization systems. In the clinical domain, the specialized information needs of the clinicians complicates the task of evaluating automatically produced clinical text summaries. In this paper we present and compare the results from both manual and automatic evaluation of computer-generated summaries. These are composed of sentence extracts from the free text in clinical daily notes { corresponding to individual care episodes, written by physicians concerning patient care. The purpose of this study is primarily to nd out if there is a correlation between the conducted automatic evaluation and the manual evaluation. We analyze which of the automatic evaluation metrics correlates the most with the scores from the manual evaluation. The manual evaluation is performed by domain experts who follow an evaluation tool that we developed as a part of this study. As a result, we hope to get some insight into the reliability of the selected approach to automatic evaluation. Ultimately this study can help us in assessing the reliability of this evaluation approach, so that we can further develop the underlying summarization system. The evaluation results seem promising in that the ranking order of the various summarization methods, ranked by all the automatic evaluation metrics, correspond well with that of the manual evaluation. These preliminary results also indicate that the utilized automatic evaluation setup can be used as an automated and reliable way to rank clinical summarization methods internally in terms of their performance.
With the large amount of information generated in health care organisations
today, information overload is becoming an increasing problem for clinicians
[
A typical situation where information overload is frequently encountered is
when the attending physician is producing the discharge summary at the end
of a care episode. Discharge summaries are an important part of the
communication between di erent professionals providing the health care services and
their aim to ensure the continuity of a patients care. However, there are
challenges with these discharge summaries as they are often produced late, and the
information they contain tend to be insu cient. For example, one study showed
that discharge summaries exchanged between the hospital and the primary care
physicians is often lacking information, such as diagnostic test results (lacking
in 33-63%), treatment progression (lacking in 7-22%), medications (lacking in
2-40%), test results (lacking in 65%), counseling (lacking in 90-92%) and
followup proposals (lacking in 2-43%) [
Computer-assisted discharge summaries and standardized templates are
measures for improving the transfer time and the quality of discharge information
between the hospital and the primary care physicians [
Automatic summarization of clinical information is a challenging task
because of the di erent data types, the domain speci city of the language, and
In: E.A.A. Jaatun, E. Brooks, K.E. Berntsen, H. Gilstad, M. G. Jaatun (eds.):
Proceedings of the 2nd European Workshop on Practical Aspects of Health Informatics
(PAHI 2014), Trondheim, Norway, 19-MAY-2014, published at http://ceur-ws.org
the special information needs of the clinicians [
We have developed an extraction based text summarization system that attempts to automatically produce a textual summary of the free text contained in all the clinical (daily) notes related to a { possibly ongoing { care episode, written by physicians. The focus of this paper is not on how the summarization system works, but rather on how to evaluate the summaries it produces. In our ongoing work towards developing this system, we have so far seven di erent summarization methods to evaluate, including a Random method and an Oracle method. The latter method representing an upper bound for the automatic evaluation score. Having a way to quickly and automatically evaluate the summaries that these methods produce is critical during method development phase. Thus the focus of this paper is how to perform such automated evaluation in a reliable and cost-e ective way.
Automatic evaluation of an extraction based summary is typically done
through having a gold standard, or \gold summary", for comparison. A gold
summary is typically an extraction based summary produced by human experts
[
To explore whether this approach to automatic evaluation is viable, we have
also conducted manual evaluation of a set of summaries, and then compared this
to the results from the automatic evaluation. A possible correlation between how
the manual and automatic evaluation ranks the summarization methods would
mean that further automatic evaluation with this approach can be considered
somewhat reliable. In this study, automatic evaluation is mainly performed
using the ROUGE evaluation package [
To produce data suited for automatic summarization, discharge summaries were extracted and associated to the daily notes they summarize. There were two types of discharge summaries: internal (written when the patient is moved to another ward and summarizing the time spent on the given ward) and nal (written when the patient leaves the hospital and summarizing the whole stay). Note that a nal discharge also summarizes any internal summaries written during the stay.
All notes and discharge summaries were lemmatized at the sentence level
using the morphological analyser FinTWOL [
The full corpus was utilized in deriving statistics about the language for some of the summarization methods. For the summarization and evaluation experiment, the corpus was narrowed down to the care episodes having I25 (Chronic ischaemic heart disease; including its sub-codes) as the primary ICD-10 code and consisting of at least 8 clinical notes, including the discharge summary. The 6 http://www.lingsoft.fi 7 http://www.nettiapina.fi/finnish-stopword-list/ latter condition justi es the use of text summarization. The data were then split into the training and test sets, containing 159 and 156 care episodes, for the parameter optimization and evaluation of summarization methods, respectively. 3
All summarization methods used in this study are based on extraction-based
multi-document summarization. This means that each summary consist of a
subset of the content in the original sentences, found in the various clinical notes
that the summary is produced from [
Random This is the baseline method, which works by composing a summary through simply selecting sentences randomly from the various clinical notes. This method should give some indication of the di cultly level of the summarization task at hand.
Oracle This is a control-method that has access to the gold summary during the summarization process. It basically tries to optimize the ROUGE-N2 F-scores for the generated summary according to the gold summary, using a greedy search strategy. This summarization method can naturally not be used in a real user scenario, but it represents the upper limit for what is possible to achive in score for an extraction based summary, or summarization method, when using ROUGE for evaluation.
The other summarization methods are here referred to as SumMet1, SumMet2, SumMet3, SumMet4 and SumMet5.
For each individual care episode, the length of the corresponding gold summary served as the length limit for all the seven generated summaries. This was mainly done so that a sensible automatic evaluation score (F-score, see Section 4.1) could be calculated. In a more realistic user scenario, a xed length could be used, or e.g. a length that is based on how many clinical notes the summaries are generated from. Each computer generated summary is run through a post-processing step, where each sentence are sorted according to when they were written. 4
We conducted and compared two types of evaluation, automatic evaluation and manual evaluation in order to evaluate the di erent summarization methods. The purpose of this study is primarily to nd out if there is a correlation between the conducted automatic evaluation and the manual evaluation. This will further reveal which of the automatic evaluation metrics that correlates the most with the scores from the manual evaluation. As a result, we would get some insight into the reliability of the selected approach to automatic evaluation. Ultimately this study can help us in assessing the reliability of this evaluation approach, so that we can further develop the underlying summarization system.
In the automatic evaluation we calculated the F-score for the overlap between the generated summaries and the corresponding gold summaries using the ROUGE evaluation package. As gold summaries we used the original discharge summary written by a physician. This gold summary is thus considered to be the optimal summary8, so we assume that it to always have an F-score of 1.0.
The conducted evaluation can be classi ed as so called intrinsic evaluation.
This means that the summaries are evaluated independently of how they
potentially a ect some external task [
ROUGE metrics, provided by the ROUGE evaluation package [
These metrics are all based on nding word n-gram co-occurrences, or overlaps, between a) one or more reference summaries, i.e. gold summary, and b) the candidate summary to be evaluated.
Each metric counts the number of overlapping units (the counting method of which depends on the variant) and uses that information to calculate recall (R), precision (P ), and F-score (F ). The recall is the ratio of overlapping units to the total number of units in the reference while the precision is the ratio of overlapping units to the total number of units in the candidate. The former 8 This is of course not always the truth from a clinical perspective, but we leave that to another discussion. describes how well the candidate covers the reference and the latter describes the quality of the candidate. The F-score is then calculated as
F =
The evaluations were performed with the lemmatized texts with common stopwords and numbers removed. 4.2
The manual evaluation was conducted independently by three domain experts. Each did a blinded evaluation of the eight summary types (seven machine generated ones plus the original discharge summary) for ve care episodes. Hence, the total sum of evaluated summaries per evaluator was 40. All summaries were evaluated with a 30-item schema, or evaluation tool (see Table 1). This tool was constructed based on the content criteria guideline for medical discharge summaries, used in the region where the data was collected. So each item correspond to a criteria in this guideline. In this way, items were designed to evaluate the medical content of the summaries from the perspective of discharge summary writing. When evaluating a summary, each of these items were evaluated on a 4class scale from -1 to 2, where, -1 = not relevant, 0 = not included, 1 = partially included and 2 = fully included. The evaluators also had all the corresponding clinical notes at their disposal when performing the evaluation.
The items in our tool are somewhat comparable to the evaluation criteria
used in an earlier study of evaluating neonate's discharge summaries, where
the computer generated discharge summaries using lists of diagnoses linked to
ICD-codes [
The evaluators experienced the manual evaluations, following the 30-item tool, to be very di cult and extremely time consuming. This was mainly due to the evaluation tool, i.e. its items, being very detailed and required a lot of clinical judgement. Therefore, for this study, only ve patient care episodes and their corresponding summaries, generated by the summarization system, were evaluated by all three evaluators. This number of evaluated summaries are too small for generalization of the results, but this should still give some indication of the quality of the various summarization methods in the summarization system. The 30 items in the manual evaluation tool are presented in Table 1. 4.3
In order to test whether the di erences in the automatic evaluation scores
between the di erent summarization methods were statistically signi cant, we
performed the Wilcoxon signed-rank test [
To identify which of the automatic evaluation metrics that best follows the
manual evaluation, Pearson product-moment correlation coe cient (PPMCC)
and Spearman's rank correlation coe cient (Spearman's rho) [
The results from the automatic and the manual evaluations are presented in Table 2. The scores from the automatic evaluation are calculated from the average F-scores from the 156 test care episodes, while the results from the manual evaluation are the average scores from a subset of ve care episodes (also included in the automatic evaluation), all evaluated by three domain experts. The latter scores have all been normalized by dividing the scores of the highest ranking method. This was done in an attempt to scale these scores to the F-scores from the automatic evaluation.
All automatic metrics and the manual evaluation agreed in terms of what summarization method belongs to the top three, and the bottom four. When calculating signi cance levels for the automatic evaluations for the ve highest ranked methods, the di erences were always signi cant. However, in several cases the di erences between the three lowest ranked methods, those being SumMet4, SumMet3 and Random, were not statistically signi cant. These results are in agreement with the fact that all the evaluation measures agreed on which the ve best performing methods were, whereas the three worst methods are equally bad, all performing on a level that does not signi cantly di er from the approach of picking the sentences for the summary randomly.
For the manual evaluation, the original discharge summaries scored signi cantly higher than any of the generated summaries. Furthermore, the summaries produced by the Oracle, SumMet2 and SumMet5 methods were evaluated to be signi cantly better than those produced by the four other methods. Thus, the manual evaluation divided the automated summarization methods into two distinct groups, one group that produced seemingly meaningful summaries, and the other that did not work signi cantly better than the Random method. The divi8 The original discharge summary is of course not a product of any of the summarization methods. sion closely agrees with that of the automated evaluations, the di erence being that in the manual evaluation also SumMet1 ended up in the bottom group of badly performing summarization methods.
In Table 3 are the PPMCC and Spearman's rho results, indicating how each automatic evaluation metric correlates with the manual evaluation scores. Spearmans rho is a rank-correlation measure, so it does not nd any di erence between most of the measures, since they rank the methods in exactly the same order (except ROUGE-SU4, which has a single rank di erence compared to others). In contrast, PPMCC measures the linear dependence taking into account magnitudes of the scores in addition to the ranks, so it observes some extra di erences between the measures. This shows that ROUGE-SU4 has the best correlation compared to the manual evaluation. Figure 2 illustrates the normalized manual evaluation scores with the ROUGE-SU4 F-scores.
All automatic evaluation metrics and the manual evaluation agreed that the top three automatic summarization methods signi cantly outperform the Random method. These methods are SumMet2, SumMet5 and Oracle. Thus we can with a certain con dence assume that this re ects the actual performance of the utilized summarization methods. Oracle is of course not a proper method, given that it is \cheating", but it is a good indicator for what the upper performance limit it.
The reliability of the manual evaluation is naturally rather weak, given that only ve care episodes were evaluated by the three evaluators. The ndings of the manual evaluation are not generalizable due to the small number of care episodes evaluated. Therefore, more manual evaluation is needed to con rm these ndings.
On a more general level, the results indicate that the utilized approach { using the original discharge summaries as gold summaries { is a seemingly viable approach. This also means that the same evaluation framework can potentially be transferred to clinical text in other countries and languages who follow a similar hospital practice as in Finland.
The manual evaluation results show that the various summarization methods are less discriminated in terms of scores when compared to the automatic evaluation scores. We believe that this is partly to blame for the small evaluation set these scores are based on, and also because of the evaluation tool that was utilized. For these reasons we are still looking into ways to improve the manual evaluation tool before we conduct further manual evaluation. It is interesting to see that SumMet2 is considered to outperform the Oracle method, according to the manual evaluators. 6.1
The agreement between the evaluators in the manual evaluation was calculated
with the 40 di erent summaries evaluated by each of the three evaluators. The
ICC value for the absolute agreement was 0,68 (95% CI 0,247-0,853, p<0,001).
There is no de nite limit in the literature on how to interpret ICC values, but
there are guidelines that suggest that values below 0.4 are poor, values from
0.4 to 0.59 are fair, values from 0.6 to 0.74 are good and values from 0.75 to
1.0 are excellent in terms of the level of interrater agreement [
Furthermore, the evaluators reported that the manual evaluation was di cult and very time consuming, due to the numerous and detailed items in the manual evaluation tool. They also reported that the judgement process necessary when evaluating the summaries was too demanding. It became obvious that several of the items in the evaluation tool were too speci cally targeting structured information. This means information that is already identi ed and classi ed in the health record system, which does not need to be present in the unstructured free text from where the summaries are generated. Examples are 'Care period', 'Care place' and 'Sender'. In the future, a shorter tool, i.e. less items, with stricter criteria and more detailed guidelines for the evaluators is needed. One important property of such a new tool would be, when used by the human evaluators, that good and bad summaries (i.e. summarization methods) are properly discriminated in terms of scoring. 7
In this paper we have presented the results from automatic and manual evaluation of seven di erent methods for automatically generating clinical text summaries. The summary documents was composed from the free text of the clinical daily notes written by physicians related to patient care.
Seven automatic summarization methods were evaluated. For the automatic evaluation the corresponding original discharge summaries were used as gold summaries for doing the automatic evaluation. Among these summarization methods were the control-methods Random and Oracle. Four ROUGE metrics were used for the automatic evaluation, ROUGE-N1, ROUGE-N2, ROUGE-L and ROUGE-SU4.
The evaluation results seem promising in that the ranking order of the various summarization methods, ranked by all the automatic evaluation metrics, correspond well with that of the manual evaluation. These preliminary results indicates that the utilized automatic evaluation setup can be used as an automated and reliable way to rank clinical summarization methods internally in terms of their performance.
More manual evaluation, on a larger sample of care episodes, is needed to conrm the ndings in this study. In this context, more research is needed to make a manual evaluation tool that better discriminates good from bad summaries, as well as being easier to use by evaluators. This preliminary work provided us good insight and ideas about how to further develop the manual evaluation tool, suited for a larger-scale manual evaluation.
As future work, we also plan to conduct so called extrinsic evaluation of the summarization methods, meaning that the various summaries produced by the system are evaluated in terms of their impact on clinical work. 8
This study was nancially supported partly by the Research Council of Norway through the EviCare project (NFR project no. 193022), the Turku University Hospital (EVO 2014), and the Academy of Finland (project no. 140323). The study is a part of the research projects of the Ikitik consortium (http://www.ikitik.fi).