This working note describes our approach to CLEF 2014 task 2a. It also reports our experimental results and discusses some future work we want to explore.
The CLEF 2014 task 2a 1 aims to provide the normalized value for each of 10 attributes of each disease/disorder mention template. Each mention template consists of the mention's Uni ed Medical Language System concept unique identi ers (CUI) and mention boundaries, and a pointer to the report from which that mention template is extracted.
Each disease/disorder mention has the following 10 di erent attributes: Negation Indicator, Subject Class, Uncertainty Indicator, Course Class, Severity Class, Conditional Class, Generic Class, DocTime Class, Temporal Expression, and Body Location. The normalized value for any of the rst nine of the attributes comes from a list of possible values, such as \yes", \no" for Negation Indicator. Normalized values for the tenth attribute-Body Location-come from the UMLS CUIs.
We decompose the task into 10 sub tasks, and consider each sub task as a classi cation problem, and accordingly design a classi er for each of the 10 attributes. We nd that for some attributes, using simple rules, e.g., always setting it to some value, yields better results than statical learning. Therefore, in addition to developing some classi ers based on machine learning, we also build some rule based classi ers. Most of the rules are automatically extracted from the annotated training data. Furthermore, we manually create a few rules, for example, some regular expressions to identify time expressions.
The main goal of our experiments is to gure out the e ective features or
rules for each attribute extraction task. In our rst attempt, we focus on such
features and rules that can be directly extracted from the training data, not
using any additional resources, such as WordNet [
For some attributes, such as Generic Class and Negation Indicator, our system works well. But for some attributes, such as DocTime Class, it performs bad. We hope to better understand the reasons once the test data with the ground truth is released. 2
In our experiments, we design a classi er for each attribute. With cross validation on the training data, we nd in most cases statistical classi ers achieve the best performance than rule bases systems. However, for \Body Location", since there are many possible values, a statistical classi er does not work. We also nd that for some attribute, such as \Course Class", always setting it to a default value gives the best performance.
In what follows, we describe details of each classi er. Note that for each attribute, the type of classi er and the features sets are determined with 10 fold cross validation on the training corpus. 2.1
We train a classi er with LIBLINEAR 2. While training, we set the type of solver to L1-regularized logistic regression, i.e., -s 6, and the regularization and experimental loss trade-o parameter to 3, i.e., -c 3.
We consider the following features in a text windows of size 17 with the mention in the middle: 1) uni-gram on the left/right of the mention; and 2) bigram on the left/right of the mention. For each n-gram, we append \:L" (\:R") to its end, indicating it is on the left (right) of the mention.
On the o cial test data, its accuracy is 0.922. 2.2
We train a classi er with LIBLINEAR with -s 6 and -c 3. We use only uni-gram in a text windows of size 17 with the mention in the middle. For each uni-gram, we also consider its position. For example, feature \weight:3" (\weight:-3")means \weight" is on the right (left) of the mention, and there are two words between them. On the o cial test data, its accuracy is 0.611. 2.3
We train a classi er with LIBLINEAR with -s 6 and -c 3. We use two sets of features in a text windows of size 17 with the mention in the middle: uni-gram with position, the same features as used for Negation Indicator Classi er; and bi-gram features with \:L" or \:R" as su x. On the o cial test data, its accuracy is 0.923. 2 http://www.csie.ntu.edu.tw/ cjlin/liblinear/ 2.4
We always set its value to \true". On the o cial test data, its accuracy is 0.961. 2.5
We build a rule based system for this attribute. First we extract all clue words for each severity class. For example, for \SEVERE", we get clue words like \acute",\sharp",\critical". Then we consider all words (context words) in a text window of size 5 with the mention in the middle and select the severity class that consists of the greatest number of clue words that appear in the text window. If there is a draw, we randomly choose one from them. And in case there are no rules to apply, we use the default value.
On the o cial test data, its accuracy is 0.611. acute severe,sharp,rapid,signi cant,critical, ash,abrupt acute onset,severely moderate to severe,signi cantly greater, pleuritic, moderately severe acute to subacute subacute, mild,extremely,advanced, high grade, profound extreme moderately to severely,acutely,markedly,high-grade,crushing severity marked,moderate-severe,more marked,extensive sharp or knife-like,moderate-to-severe sub acute moderate to large sized, breakthrough, substantial coarse, massively critically high, sudden onset,volatile massive,considerable MODERATE moderately,mildly,mild,moderate,modest,mild to moderate,dense mild-moderate,markedly,mild-to-moderate,moderate to large subacute,signi cant,mildy,large-to-moderate UNMARKED worsening,severity,elevated,increase SLIGHT trace,mild,slight,minimal,minimally,minimally displaced mildly,slightly,modest,minor,partially,much lesser extent minimal amounts,minimal amount,trivial,partial,min,little 2.6
We always set its value to \false". On the o cial test data, its accuracy is 0.936. 2.7
We train a classi er with LIBLINEAR with -s 6 and -c 3. We use only uni-gram with position as features in a text windows of size 17 with the mention in the middle. On the o cial test data, its accuracy is 1.000. 2.8
Since there are potentially many body location labels, a statistical classi er with a xed number of labels will not work. Therefore, we build a rule based classier for this attribute. Each rule is is a clue word body location pair, which are automatically extracted from annotations. Here are some examples: (inferoapical,C1299408), (liver,C0223884). The procedure of extracting rules is similar to what we have done for Severity Class. With the mined rules, we select the body location that has the greatest number of clue words that occur in the mention. In case of a tie, we randomly choose one from them. On the o cial test data, its accuracy is 0.635. 2.9
We always set its value to \OVERLAP". On 10-fold cross validation, it
outperforms Support Vector Machines (SVMs) [
We build a rule base classi er for this attribute. We run two steps to construct the rules: 1) rst we extract clues words for each \Temporal Expression", i.e., DATE, DURATION and TIME; and 2) we manually compile regular expressions based on the clue words to make them more general. Step 2 is necessary because that clue words related to temporal expression are often long, and contain concrete numbers, making them hard to be matched. Here are some examples of such rules: DATE ndf4g-ndf2g-ndf2g, DURATION ((several) j (nd+) j (one)j(two)j(three)j(four)j( ve)j(six)j(seven)j(eight)j(nine)j(ten))ns+((minute) j(second)j(hour)j(day)j(week)j(wk)j(month)j(year)j(yr)j(mn))s?ns+((duration) j(interval)j(history)), where \DATE",\DURATION" are labels. In total, we have 30 regular expression based rules.
To do the prediction, we consider a text window of size 11 with the mention in the middle, to which we apply all the compiled regular expressions. Finally we choose the label that has the greatest number of matched regular expressions. In case of a tie, we randomly choose one; in case not any regular expression is matched, we choose the default value, i.e., \none".
On the o cial test data, its accuracy is 0.824. 3
We build 10 classi ers to handle Task 2a. These classi ers can be organized into three groups: the rst group of classi ers always predict the same value. This strategy works very well for Course Class and Conditional Class, but does not work for DocTime Class; the second group of classi ers are based on rules mined from clue words of disease/disorder mentions. The rules are used for majority based voting to output the prediction. Classi ers for Severity Class, Body Location and Temporal Expression belong to this group; the third group of classi ers are based on SVMs and mainly use n-gram features extracted from a text window with the mention in the middle. There are 4 instances in this group, i.e., classi ers for Negation Indicator, Subject Class, Uncertainty Indicator and Generic Class. Except for the classi er for Subject Class attribute, the classi ers in this group perform well with an accuracy of more than 0.900 on the o cial test data.
In our current experiments, we don't use any external resources, and don't use any features more complicated than n-grams. In future we would like to consider more resources and more advanced features, such as features from the values of related attributes, particularly for the attributes on which our current approach does not work well.