The CLEF eHealth 2018 Task 1 is aimed to automatically assign ICD10 codes to the text content of death certi cates. The challenges of this task is that participants have to extract information from written text in unexplored French language corpora, which means that all these ICD10 codes have little data used to train. In this paper, our team proposes some methods to solve the Task 1. We utilize two machine learning method, Xgboost and RandomForest, meanwhile, we also take advantage of some association rules and similarity computation to boost the performance of our method. We evaluate our results using the evaluating code provided by organizer.
Based on the pre-work of the 2016[
The data set is called the CepiDC Causes of Death Corpus, it comprises free-text descriptions of causes of death which are reported by physicians in the standardized causes of death forms. The training data have two types: raw data and aligned data. Raw data contains 65,843 death certi cates, and di erent les have di erent information, such as DocID, RawText, Gender, Age and so on. Di erent from raw data, aligned data combines causes information, identi cation and cause labels together. Therefore, every case in aligned data has complete information, and the last three elds are CauseRank, StandardText and ICD10, which will not exist in test data.
A B J I
Z
Text septicemie streptocoque B
hemolytique septicemie streptocoque B septicemie
streptocoque alpha-hemolytique . . . septicemie streptocoque septicemie staphylocoque
catheter dialyse ... sepsis ORL sepsis origine
pulmonaire sepsis mediastino-pleural sepsis
mediastinal ... senilite vasculaire senilite cardio-vasculaire senilite
cardiaque ... sedation antiepileptique sedation antidouleur sedation antalgique ...
26
In this task, the organizers have provided vague tag and the causes of death are all medical vocabulary. The ICD10 codes, such as S299, V892 and I259, are in the same category. In this way, it is less useful to classify those data depending on semantic information. In order to obtain accurate results, we propose two methods: Regular Match Expressions and hybrid approach based on machine learning. We should assign one or more ICD10 codes to each cause of death (one case may contain several ICD10 codes). In one le, there are more than 3,000 ICD10 codes. The dictionaries are summarized in table 1(6 versions of a manually curated ICD10 dictionary developed at CepiDC).
Our system architecture is showed in Figure 1. We mainly utilize the Regular Match Expressions to obtain the results. In order to handle those data which are unable to nd out the mapping expressions, we extract some features from training data and utilize machine learning method to classify. To improve the accuracy rate, we design a strategy to pre-classify these aligned data into 26 tags whose range is from A to Z and those tags represent the rst code of ICD10 codes. Furthermore, we apply similarity computation and regular match expressions to obtain the digital code behind the alphabet code. At last, we combine the results of machine learning classi cation and regular classi cation as the nal results.
Xgboost
RandomForest
Similarity Computation
Regular Match
Expressions Machine learning classification Combine
Runs
Regular Match Expressions
Regular classification The language of data is French, therefore, it is so di cult for us to understand those causes of death perfectly. According to the requirement of this task, we are prohibited from using translation tools, thus, what we do is try our best to extract the features and inner connection between data. And the majors of the members of our team are all computer science, so we know little about medical knowledge. What's more, in the case of data itself, the di erence between di erent categories is not obvious. 3
We extract raw text, standard text and dictionaries provided by the organizers in training data as gold data. We divide gold data into 26 sets fset A, set B, . . . , set Zg according to the algebraic code, for example, the raw text set is fS1, S2, S3, S4, S5g and the corresponding ICD10 code set is fA123, A234, D135, X246, A145g, therefore, fS1, S2, S5g belongs to set A and fS3g belongs to set D, and fS4g belongs to set X. Besides, we use Google's word2vec model to train the whole gold data to get the vec.bin le in order that we have an access to compute similarity between training data and test data. We extracted the ICD10 corresponding code from the training data provided by the organizer and extracted all the raw text and aligned texts as corpus, and then directly used google's word2vec model to train our own corpus. At the same time, because of the grammatical and formal di erences between French and English, we need to do some preprocessing, such as extracting stems and removing stop words. In addition, we did not consider the di erence between English and French too much, because word2vec does not need any known semantic knowledge during training, and it completely depends on the corpus. 3.1
We regard gold data as regular expressions to match, so if the text T in test data directly appears in gold data, we predict that the ICD10 code of T is the corresponding code. However, sometimes T may contain two or more kinds of causes of death, in this case, we will split T according to expressions. For instance, T is \Tableau de mort subite au cours d'un e ort sportif" and regular expressions R contains R1\mort subite" and R2\e ort sportif". When we use T to match R, we will nd that R1 is included in T , so we will predict that the ICD10 code of T is the same with R1. This method ignores R2 and lead to an incomplete prediction, so we will split T into two parts, T1" mort subite" and T2" Tableau de au cours d'un e ort sportif", and use T2 to match R and split T2 until there is no more match between Tn and R. In the end, the ICD10 codes of T is a collection of all Ti's mapping codes.
Because there are two or more causes of death in a raw text, we adopted an iterative strategy to improve the accuracy of the regular match: For S belonging to the raw text set, we set it with the regular expression set R = fr1; r2; :::rng to match, when a certain expression ri is matched in S, ri is extracted from S as T1, and the remaining element in S is taken as S; judging whether S is empty, if it is not empty, it continues to match R and split it and record ri as Ti,until the nal phrase or word(entities) S can't nd matching elements in R. At this time we consider the match is over. 3.2
We utilize two kinds of machine learning methods: bagging and boosting.
RandomForest[
Age death certi cate ID year the death certi cate was processed by CepiDC gender of the deceased age at the time of death, rounded to the nearest ve-year age group
length of time the patient had been su ering from coded cause
assigned by coder dictionary entry or exerpt of the raw text that supports the selection of an
gold standard ICD10 code
1 2006 1/0 35 1 => Home 2 => Hospital 3 => Private Clinic 4 => Hopice, Retirement home 5 => Public place 6 => Other Location if the patient had been experiencing the cause for 6 months, "IntValue" should be 6 and "IntType" should be 4 6-1 surinfection
J969 hemorragie digestive 3.3
We obtain 26 sets according to the algebraic code of the test data using machine learning method and then we apply similarity computation in each set between test data and training data, similarity computation is a method to measure how two words are close to each other in semantic meaning, for example, we get a set S1 in test data which is classi ed as set A using machine learning method, then we perform similarity computation between set S1 and all text of set A in the training data, we consider the ICD10 code of S1 same as training data where the maximum value obtained. 3.4
We combine our results achieved from regular classi cation and machine learning classi cation in order to obtain a perfect performance. We regard the results obtained by regular classi cation as our baseline. Then, we use the runs achieved by machine learning classi cation to supply and modify the ICD10 codes in baseline. Suppose S in training data fails to match appropriate text in regular classi cation, which means that the ICD10 code of S is empty, we treat S as an input of machine learning method and gure out the alphabet code of S. And then, we use dictionaries and standard text in training data to compute the similarity between them. Finally, we chose the most similar text and treat its mapping ICD10 code as the result of S. In this way there is no con ict between the results of regular classi cation and machine learning classi cation, since the work of regular rules is based on the classi cation of machine learning methods. 4
We utilize the les "AlignedCauses 2006-2012full.csv", "AlignedCauses 2013full.csv", and "AlignedCauses 2014 full.csv" provided by organizers to train and test our methods. We divide the data into training A set and test A set according to the ratio of 8 to 2, besides, we also regard the le of 20062013 as training B set and the le of 2014 as test B set to validate our approaches. Speci cally, we submit two runs based on two methods, where the description for each method is as follows.
Method (A). We utilize machine learning methods rst; in which we extract semantic information. In order to avoid the in uence of noise information, we use some NLP tools to delete the stop words and do some stemming works. What's more, we manually set some feature sets and nd out which set is able to get the most accurate results. And then, we divide training A into 26 types and each type contains the data with the same alphabet, for example, the beginning of ICD10 codes of data in type A is all A. Finally, we use 26 types data to match regular expressions, compute the similarity, and predict the nal results of test A.
Method (B). We utilize rules mainly based on regular match expressions to extract the ICD10 codes. It means that if the raw text T in test B set is matched with the raw text R or standard text S in taining B set, we think the T and R or S have the same ICD10 codes. Because the raw text T in test B may contain many new written text which haven't appeared in training data, we apply machine learning methods to handle the mismatching data. We select DocID, YearchCoded, Gender, Age, LocationOfDeatch, LineID, RawText, IntType and IntValue as features to train RandomForest and Xgboost model to pre-tag each raw text in test B. After pre-tagging, we use similarity computation or regular expressions to predict the complete codes.
The primary evaluation measure of this task is the precision, recall and F1. The organizers provide participants with the evaluation program, thus we should use standard program to evaluate our runs.
0.4368 0.4368 0.4504 0.5808 0.5445
0.4555 0.0 0.2005 0.4102 0.475
0.5577 0.5577 0.4511 0.6343 0.6407
0.5777 0.0001 0.2525 0.5066 0.579
ECNUica-run2 frequencyBaseline moyenne mediane 0.7712 0.7712 0.4517 0.7123 0.7712 FR raw-ALL
0.7895 0.1 0.341 0.7228 0.7981
In 2018 CLEF eHealth task 1, we propose a regular match expression method and utilize machine learning methods and similarity computation to improve the accuracy of the prediction of the ICD10 codes. However, we still have some problems to solve. The features we select from training data are some normal features, such as age, gender and raw text. In the future, we will pay more attention on the research of extracting useful features and discovering the inner connection of raw data to train machine learning methods. 6
Suominen, Hanna and Kelly, Liadh and Goeuriot, Lorraine and Kanoulas, Evangelos and Azzopardi, Leif and Spijker, Rene and Li, Dan and Neveol, Aurelie and Ramadier, Lionel and Robert, Aude and Zuccon, Guido and Palotti, Joao. Overview of the CLEF eHealth Evaluation Lab 2018. CLEF 2018 - 8th Conference and Labs of the Evaluation Forum, Lecture Notes in Computer Science (LNCS), Springer, September, 2018. Neveol A, Robert A, Grippo F, Lavergne T, Morgand C, Orsi C, Pelikan L, Ramadier L, Rey G, Zweigenbaum P. CLEF eHealth 2018 Multilingual Information Extraction task Overview: ICD10 Coding of Death Certi cates in French, Hungarian and Italian. CLEF 2018 Evaluation Labs and Workshop: Online Working Notes, CEUR-WS, September, 2018. 7. A. Liaw, M. Wiener et al., \Classi cation and regression by randomforest," R news, vol. 2, no. 3, pp. 18{22, 2002. 8. T. Chen, T. He, M. Benesty et al., \Xgboost: extreme gradient boosting," R package version 0.4-2, pp. 1{4, 2015. 9. T. Chen and C. Guestrin, \Xgboost: A scalable tree boosting system," in Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. ACM, 2016, pp. 785{794.