The CLEF eHealth 2016 Task 1 is set to automatically assign pre-de ned medical tag to each word in the patient case records. The di culty of the task is that many classes have little training data. This paper presents our work on the 2016 CLEF eHealth Task 1. In particular, we propose an optimized Conditional Random Field algorithm to better ful ll the task. We also utilize the information extracted through association rules and MetaMap to boost the performance of our results. The evaluation results show our runs outperform the four o cial baselines in this di culty task.
CLEF eHealth 2016 Task 1 addresses clinical information extraction, related to
Australian nursing shift change[
The data set utilized in this task is called NICTA Synthetic Nursing Handover
Data [
In this task, the organizers provide us with 36 tags which are related to the categories of Patient Introduction, My Shift, Appointments, Medication and Future Case in the handover form. We are asked to assign one of the 36 tags to each word in the patient case records. The 36 tags are summarized in table 1.
Our system architecture is proposed in Figure 1. We mainly utilize the
Conditional Random Field (CRF) model to achieve the results [
The organizers provide us with 17 features for each word in the health care
records, which are listed in table 2 [
Suppose A = ffig; i = 1; 2; :::; 17 is the feature set which contains the whole 17 features, and B is the feature set which contains features should be removed from A. For a tag, such as PatientIntroduction CarePlan, we utilize the following method to discover its suitable feature set for the CRF model training. Denote C as the suitable feature set for the PatientIntroduction CarePlan tag. { Utilizing feature set A to train CRF model in data set 1, and using data set 2 to test the model. denote the precision of PatientIntroduction CarePlan as p. { 8fi 2 A; i = 1; 2; :::; 17, remove fi from A. Train the CRF model by utilizing the data set 1 based on the feature set A without fi. Test the learned CRF model on data set 2. Denote the precision on the PatientIntroduction CarePlan tag as pfi . { Calculate the mean value of pfi ; i = 1; 2; :::; 17 and denote it as pm. If pfi > p and pfi pm > 10%, then put the feature fi into set B.
NER
4 (PpOarSt of speech)
6 Basic dependents Bwoarsdic dependents of the 9 Top 5 candidates vTeodp f5rocmanUdiMdaLtSes retrie10 Top mapping
in ATCL (Anatomical The11 Medication score rapeutic Chemical List); else 0.5 if it can be found in ATCL; 0 otherwise
12 Location to its end
tenpoint scale from the \1" for the rst word beginning of the document and \10" for the last NICTA word
term occurs in a document 13 Normalized term divided by the maximum frequency of this term frequency over all terms in the document
14 Top 5 candidates Nomenclature of Medicine
15 Top mapping
16 Top 5 candidates As 9 using AMT5 17 Tom mapping
{ The suitable set C = A B. This means that the features in the set B will decrease the precision and ought to be removed from the features collection.
Thus, we obtain a suitable feature set for each tag. The suitable feature sets are noted as Ci; i = 1; 2; :::; 36, corresponding to the 36 medical tags. Next step, we utilize Ci; i = 1; 2; :::; 36, to train CRF model respectively, and we obtain 36 results by using those learned CRF model to perform labelling task in a patient case record. Then, we combine these results through the method of voting. Note the 36 medical tags as ti; i = 1; 2; :::; 36. For a word w in the patient case record, suppose the tag ti appears ni times. We select the tag which have the highest voting count as the nal tag of the word w. Suppose tA is the label of w obtained by the CRF model which is trained by utilizing the feature set A. If two or more tags which share the same highest appear time we select tA as the nal tag of w. 2.2
We analyze the test and training set and discover that information about patient name, room, bed, gender, age and doctor name have the same expressing format. Thus, we utilize association rules to extract relevant information and assign them with the relevant tags automatically. The association rules are listed in table 3 2.3
We use MetaMap6 to obtain the Uni ed Medical Language System (UMLS) concepts in the patient case records. Meanwhile, we identify the words with MetaMap tag \phsu" and assign them with the tag \Medication Medicine". 2.4
We combine our results achieved from the CRF model, association rules and MetaMap in order to get a better performance. Runs obtained by CRF models are utilized as our basic runs. We use the results achieved by using rules and MetaMap to modify the tags in CRF runs.
Suppose SC ; SR; SM are the results achieved by CRF, association rules and MetaMap respectively. For a word in a patient case record, if its tag in CRF run is di erent from its tag in rule and MetaMap runs, we utilize the tag in association rules and MetaMap runs to replace the tag in CRF run. Note that, there is no con ict between the tags resulted from the association rules and the MetaMap, since the two methods extract information for di erent tags. 6 https://mmtx.nlm.nih.gov/ word in fmale,him,his,he,gentleman, gentlemen, man,men,boy,boys,himself g or ffemale,her,she, hers,lady,ladies, woman,women,girl,girls,herself g without the word \under" + words PatientIntroduction GivenNames/Initials capitalized the rst letter
without the word \under" + words capitalized the rst letter \under" + (\Dr") + name
We utilize the features in the le \CRF Matrix noLabel.data" provided by organizers to train our CRF model. We implement the training and tagging process of CRF by a open-source toolkit named \CRF++-0.58". Data set 1 and data set 2 are all used for training a CRF model to tag each word in the data set 3. Speci cally, we submit six runs based on two methods, where the description for each method is as follows.
{ Method A: We use rules mainly based on regular expressions to extract information of bed number, room number age and doctor's name. Then, we use all features provided to run CRF model and obtain label for each word. Finally, we combine the result achieved by CRF model with result obtained by the association rules. { Method B: We use the method, which is detailed in section 2.1, to select the best suitable feature set for di erent label among the features provided by the organizers. Then, we train the CRF model based on those feature sets. At last, we use the method of voting to determine the label for each word. We also use association rules to extract information of bed number, room number, age and doctor's name. The nal submission is the combination of results obtained by the association rules and the voting methods.
The primary evaluation measure of this year is the macro-precision(MaAPrec), macro- recall(MaARec), macro-F1(MaAF1), micro-precision(MiAPrec), microrecall(MiARec), micro-F1(MiAF1), precision of NA tag(NA-Prec), recall of NA tag(NA-Recall), F1 value of NA tag(NA-F1). Data set 1 is used for training. Data set 2 is used for validation and data set 3 is designed for test. Evaluation of our methods over the data set 1, 2 and 3 is summarized in Table 3, where A and B imply our method A and method B respectively. 4
In 2016 CLEF eHealth task 1, we propose an optimized CRF model and utilize the association rules and MetaMap tag to achieve the better performance for the handover form automatically lling. All of our submissions outperform the four baseline methods. In the future, we will continue on the research of handover form automatically lling methods. 5
This research is funded by National Key Technology Support Program (No. 2015BAH12F01-04) and Science and Technology Commission of Shanghai Municipality (No.14DZ1101700). We also thank anonymous reviewers for their review comments on this paper.