Clinical Named Entity Recognition is a part of Task 1b, organised by CLEF eHealth organisation in 2015. The aim is to automatically identify clinically relevant entities in medical text in French. A supervised learning approach has been used for training the tagger. For the purpose of training, Conditional Random Fields(CRF) has been used. An extensive set of features was used for training. Precision, recall and F1 Score were used as evaluation metrics. Ten fold cross validation technique was used to evaluate the system. The best precision obtained was 0.91 and the best recall obtained was 0.66. After the test results were announced, the best F1 score obtained for exact matching was 0.67 and for relaxed case (i.e. inexact matching), it was 0.73.
There is a huge amount of raw medical data available in the form of textual information. The goal of Information Extraction, here, is to present the data in a way that enhances user experience and allows better comprehension. The major task involved in this process is the identi cation of named entities within the document. This allows the user to have a better understanding of the jargons. It also allows to identify important terms that may be helpful in summarising the medical document.
The Clinical Named Entity Recognition is di erent from other common sequence tagging problems, like POS (Part of Speech) tagging. The major point of di erence is the existence of ambiguity in the medical document. The span of an entity may overlap with the span of another entity i.e. the same word can be a part of multiple entities. Another issue with it is the presence of non contiguous entities. That is, the span of the entity may be discontinuous over a sentence. Moreover, there are ample resources such as thesaurus, but almost all of them are English centric. The training data, being in French language, poses another challenge.
In order to annotate the clinical entities, UMLS (Uni ed Medical Language System) is used. It is a compendium of vocabularies in biomedical science. There are various semantic groups, under which an entity can lie.
In order to tackle the problem, an extensive list of features were used. CRFsuite software was used to train the tagger. The following sections explains in greater details the methods and tools used for tackling the problem. 2 2.1
The available data was rst pre-processed by stemming all the words. We have considered the NER task as a sequence tagging problem. We, therefore have used CRF basic tagger to train the system. CRF is a state of the art method used for the purpose of sequence tagging. We have used CRFsuite software for this purpose. 1. Lexical Features: Uni-grams, bi-grams and tri-grams of words were used as features within a window of 3 around the current word. Then the POS tags of the words in the window were also chosen as features. Finally, we used capitalisation of letters and presence of digit as features too. In addition, pre xes and su xes of 3 letter length were also used as features for each word. 2. UMLS Features: UMLS features were extracted using MetaMap. Since, MetaMap is English-centric, therefore, all the French words in the training data were translated into English, separately. Then, using MetaMap API, semantic group of these words were obtained and used as features for training. 3. Global Features: For every word, we calculated the position of the word in that sentence. To do this, we treated the word as LEFT, when it lied in the left quarter of the sentence. When it lied in the right quarter of the sentence, it was treated as being RIGHT. Otherwise, it was treated as being CENTER.
In order to account for the case when the span of entities was over multiple words in a contiguous manner, we used BIO format. For the starting of an entity, its name was pre xed with -B. If it was an intermediate word, the entity name was pre xed with -I. Otherwise, it was named as O. The system currently does not handle the discontinous terms. 3
Tools Used 1. Microsoft Bing translator was used for translating each french word (separately) into English. 2. Snowball stemmer was used for the stemming during pre-processing step 3. CRFSuite tagger was used to train the model based on the traing data and for tagging the test les. 4
The training data was provided by the CLEF organisation itself. The data consisted of 833 MEDLINE documents, that contained single lines of medical data in French language. An additional 11 EMEA documents were also provided. Annotation les for each document were also given.
Some statistics of the training data is as follows:
Total Word Count 25,500
Number of Annotations 5,690 Non Contiguous Annotations 40 Overlapping Spans of entities 797
As can be seen, the non contiguous annotations account for just 0.7% of the total number of annotations and hence don't a ect the validations, much. 5
Precision, Recall and F1 measure were used as evaluation metrics to evaluate the system. These are de ned as follows:
T rueP ositives P recision = T rueP ositives+F alsepositives
T rueP ositives Recall = T rueP ositives+F alseNegatives
F 1Score = 2 P recision Recall
P recision+Recall
We used ten fold cross validation techniques. The available data was partitioned into training data and validation data. The partition was done randomly, i.e. random samples were taken from the data and used for validation. The partition was done four times in the ratio of 60:40, 70:30, 80:20 and 90:10. Ten di erent and random partitions for each ratio were created. Then, the average precision and recall was calculated.
When MetaMap was not used as an external source of information, following results were obtained: 1. Run 1: Predictions were completely made by CRFsuite based on the model generated using training data. 2. Run 2: Predictions made use of CRFsuite software as well as UMLS Metathesaurus information obtained by MetaMap. Whenever the model tagged a token as a non-entity, information from MetaMap was used to specify the tag for it. It was done at the level of single word only. The runs were submitted for entity identi cation only. We didn't participate in entity normalisation. 6.2
For the EMEA documents, following results were obtained:
Run Run 1 Run 2
Exact Match Inexact Match Precision Recall F1 Score Precision Recall F1 Score We present a supervised Clinical named Entity Recognition system that can detect the named entities from a French medical data, using an extensive list of features, with an F1 Score of 0.68. It also uses UMLS Metathesaurus information obtained by MetaMap, using the English translated version of French words.
The surprising thing to observe was that although, we used extra UMLS Metathesaurus information obtained by MetaMap in run2, although it improved the precision, but reduced the recall drastically. This remains to be investigated.
The system does not handle the non contiguous entities properly. We can use mutual information as a technique to identify the relationship between di erent entities, in order to detect that.