We present an elegant and effective approach for addressing limitations in existing multi-label classification models by incorporating interaction matching, a concept shown to be useful for ad-hoc search result ranking. By performing soft n-gram interaction matching, we match labels with natural language descriptions (which are common to have in most multi-labeling tasks). Our approach can be used to enhance existing multi-label classification approaches, which are biased toward frequently-occurring labels. We evaluate our approach on two challenging tasks: automatic medical coding of clinical notes and automatic labeling of entities from software tutorial text. Our results show that our method can yield up to an 11% relative improvement in macro performance, with most of the gains stemming from labels that appear infrequently in the training set (i.e., the long tail of labels).
Multi-label text classification (i.e., the task of assigning a variable number labels to a piece of text) is a classic task with a variety of practical applications. For instance, a clinical report could be tagged with medical codes, describing a patient’s diagnoses (e.g., Lyme disease) and procedures (e.g., clipping of aneurysm). Since both the note and the medical codes are required in the clinical process, a system that can automatically derive these medical codes from notes would be a valuable time-saving measure. As another example, software tutorial text can be semantically labeled with the tools that are used to accomplish the task. These labels could provide additional support to users trying to replicate a tutorial, or be used to improve search engines by indexing on these labels.
Text labeling approaches rely on machine learning
techniques to rank a given set of labels for a piece of text.
For example, supervised FastText
We present a text labeling approach that uses soft
ngram interaction matching, an approach inspired by recent
work in ad-hoc ranking
In summary, our contributions are: (1) an approach for extending multi-label classification models, based on soft n-gram interaction matching; (2) an evaluation on two datasets, showing that our approach can be effectively combined with other leading classification approaches; and (3) an analysis demonstrating our capacity to identify long tail labels, even those without training samples.
Multi-label text classification. This is a well-studied task
with a multitude of prior work. Among the most notable
recent efforts are supervised FastText
Soft n-gram interaction matching. Interaction-focused
ranking models formulate document ranking as a
learningto-rank problem over a term similarity matrix between a
query and a document. One successful approach to learn
these patterns is by applying square convolutional kernels
over the term similarity matrix (e.g. Pang et al. (2016);
Notation. Let T be a sequence of tokens in a document, and let L be a collection of labels. For multi-label text classification, a score V Li;T is generated for each label Li 2 L. The labels are ranked by this score, optionally employing a score threshold and/or a maximum count to select the labels.
For our task, we assume each label consists of a sequence of tokens representing its name in natural language: Li = fLi;1; Li;2; :::; Li;jLijg. This is a reasonable assumption, given that the labels are usually produced for humans, who will often need a name to reason about the label. For our method to be most effective, the names should have terms that may have approximate matches in the text. For instance, given the procedure label of clipping of aneurysm, an approximate match found in the text might be clip the aneurysm. Our method uses the term similarity matrix SLi;T 2 RjLij;jT j between label Li and text T as input:
SjL;ki;T = cos(embed(Li;j ); embed(Tk)) (1) where cos( ) is the cosine similarity score and embed( ) returns the token’s word embedding. Note that each similarity score here is a unigram match; our model operates over this matrix to perform n-gram matching. An example similarity matrix is shown in Figure 1.
Interaction matching model. Inspired by recent interaction-focused ranking models in information retrieval, we apply square convolutions over the similarity matrix to produce soft n-gram matching scores. Unlike document ranking models, however, we impose a single fixed sequential convolution kernel over the labels. In other words, we use the identity matrix IjLij as a convolutional kernel. We then take the maximum scores from each kernel and normalize them by the length of the label jLij. This step is not taken in the document ranking models but necessary in this context because multiple labels of different lengths are being matched over the same document. Note that since the convolutional kernel matches similarity scores, exact matches are not required; this is what makes the n-grams ‘soft’. More formally, our method generates a label score for each document position P Li;T 2 RjT j for document T and label Li: (2) P Li;T =
IjLij ? SLi;T
jLij where ( ) is the sigmoid activation function and ? performs 2-dimensional convolution. For simplicity, we assume ? applies padding where necessary.
To generate a label score for the entire text, we perform max pooling: V Li;T = maxjjT=j1 PjLi;T . The use of max pooling allows for the soft n-gram to match anywhere in the document and for the score not to be influenced by document length (as opposed to average pooling, for instance). Furthermore, the arg max yields an interpretable grounding of the model’s decision within the text and can be used to aid in the explanation of the model’s decision. At inference time, all labels in L are ranked using this method. This approach is trivially parallelizable and easily handles datasets with thousands of labels. An example of interaction matching scores are shown in Figure 1. In this example, the exact match is given a perfect matching score of 1.
The model’s structure allows the interaction model to easily incorporate new labels to be introduced after training simply by adding to L. In our experiments, we train the model by back-propagating errors to the word embeddings. We recognize that our model is ineffective for labels Report: ...status post tracheostomy for paradoxical vocal cord motion with asthma discharge medications fenofexadine mg po q day calcium carbonate grams po t i d percocet one po q to hours prn pain...
ICD-9 labels: Other diseases of upper respiratory tract ; Asthma ; Diabetes mellitus Tutorial: Create another new document (I chose 600 x 400 px for width and height), select the brush tool, and open the brush preset panel.
Tool labels: File > New ; Brush Tool that do not match the text. Thus, we suggest incorporating our method into existing multi-label text classification approaches, which can learn to effectively match labels that frequently occur in the training data. We train the two models jointly, combining them by taking the maximum score for each label.
We test our approach on two tasks: medical coding and software tool extraction from online tutorials. While both are multi-label classification tasks, the data characteristics are different for each task, demonstrating that our approach is generally applicable. Examples are given in Figure 2.
Dataset. We first evaluate on the MIMIC-III dataset
Baselines & training. We compare our approach to the state-of-the-art attention-based CAML (Mullenbach et al. 1https://mimic.physionet.org/; https://www.cdc.gov/nchs/icd/icd9.htm Model Bi-RNN + interaction (ours) CNN + interaction (ours) CAML + interaction (ours)
MacroP
MacroR
MacroF1
MicroF1
2018) network for medical coding, along with a
convolutional neural network text classifier network
Results. Our results on MIMIC-III are presented in Table 2. In line with prior work on the dataset, we measure the performance in terms of micro- and macro-averaged F1 score. Since our focus is improving the long tail of infrequently-occurring labels, we also include macroaveraged precision and recall. Our approach outperforms the state-of-the-art CAML method in terms of macro precision, recall and F1 by 3–5% (relative improvement, significant at p < 0:05). The performance improvement on the weaker CNN baseline is even more pronounced, achieving an 8– 11% improvement on the macro metrics (also significant at p < 0:05). Interestingly, our approach improves the precision for the bi-directional RNN, at the expense of recall. We attribute this to the interaction matching technique that is inherently high-precision. The improvements in the micro metrics are less pronounced, showing that our approach primarily benefits the long tail.
Error analysis. We often found that our method was able to match infrequent labels where CAML had failed. For instance, in one report, our method labeled all three codes correctly (including one that occurs in only 0.5% of training data), while the unmodified CAML method found two of the three correctly, but also mistakenly included a third, completely unrelated label (occurs in about 0.1% of training data). We observed cases where general codes were not +Inter.
Sentence 1 1 7
Go to Filter > Texture > Craquelure. Change the Crack Spacing to 13, the Crack Depth to 3, and the Crack Brightness to 8.
Now in your new layer, using the Radial Gradient Tool, drag a red gradient over the whole document.
Set the duration of frame 2 to .05 seconds Now it is time to create a path P. If we have our path the right click of the mouse and stroke path with brush set 10px hardness of 100%. matched effectively by either model. For instance, Other diseases of lung is difficult to match by both models because it involves more advanced reasoning (i.e., the condition affects the lung, and there isn’t another label).
Dataset. We also evaluate our approach on a collection of software tutorials, labeled with the tools used to complete each step (by sentence). This dataset is collected from online tutorials and manually labeled by sentence with a large collection of software tools (831 in total). The dataset consists of 40k sentences, with an average length of 21.7 tokens and an average number of 1.2 tools labeled per record. An example labeled tutorial sentence is given in Figure 2. Note that the tool mentions can be either explicit (brush tool ! Brush Tool) or implicit (Create a new document ! File > New). The dataset will be made available for validation of our results. We use a random 90/5/5% train/dev/test set split.
Baselines. Since no specialized systems exist for this
dataset, we use supervised FastText
Results. We present our results on the tutorial dataset in Table 4. We use Average Precision (AP, macro-averaged by label). This evaluation emphasizes correctness along the long tail (as opposed to a micro average). When applied to FastText, our approach improves the test set performance by 6% (relative improvement, significant at p < 0:01). The FastText model achieved a perfect AP score of 1.0 for 55 of the labels found in the test set (meaning it was ranked highest among all labels whenever it appeared), whereas the interaction variant had a perfect score for 69 labels. This is a 25% improvement, most of which came from the less frequent half of the labels. The least frequent quarter saw an even bigger change, from 11 perfect scores to 26 (136%), four of which had no training samples. Our approach also slightly improves the performance on XML-CNN, though the results are not significant at p < 0:01. Interestingly, the XML-CNN model appears to hamper performance in the development set. Finally, the BERT model underperforms FastText and XML-CNN. This is likely in part due to the small amount of training data available for many labels.
Qualitatively, we find that the interaction approach improves in situations in which there are similar terms/phrases in the long tail. Examples are given in Table 3. Specifically, in cases in which there is similar text to a label in the sentence, the interaction approach is beneficial (a) and (b). We acknowledge that the interaction mechanism can occasionally have false matches (c), and it does not improve performance when there is no similar text (d).
We presented an approach to enhance existing multi-label classification techniques that employs soft n-gram interaction matching. We demonstrated that the approach is effective at identifying labels in the long tail, which are underrepresented with current state-of-the-art classification approaches. We also showed that the approach can effectively label items that do not appear at all in the training data.