As an increasing number of new journal articles being added to the MEDLINE database each year, it becomes imperative to build effective systems that can automatically suggest Medical Subject Headings (MeSH) to reduce effort from human annotators. In this paper, we propose three approaches, one building upon another in an incremental way, to automatic MeSH term suggestion: 1) MetaMap-based labeling, which relies on the MetaMap tool to detect MeSH-related concepts for indexing; 2) Search-based labeling, which builds on MetaMap-based approach and further leverages information retrieval techniques for finding similar articles whose existing annotations are used for MeSH suggestion; 3) LLDA-based labeling, which further trains a multi-label classifier based on MeSH ontology for MeSH candidate list pruning. The evaluation on the BioASQ challenge data shows promising results.
MEDLINE3 is the U.S. National Library of Medicine’s (NLM) premier
bibliographic database that contains over 19 million references to journal articles in
life sciences with a concentration on biomedicine. MEDLINE records are indexed
with Medical Subject Headings (MeSH) and by highly qualified domain experts.
Currently, there are about 0.7 million new journal articles being added to the
MEDLINE databsese each year, which makes manual indexing extremely
difficult and costly. Besides, the indexing consistency among domain experts is
unpredictable and hard to control. Funk and Reid [
In order to alleviate those problems, the NLM has developed a tool called Medical
Text Indexer (MTI) to assist human annotators with MEDLINE article
indexing [
In this paper, we propose three approaches, one building upon another in an incremental way, to automatic MeSH term suggestion: 1) MetaMap-based labeling, which relies on the MetaMap tool to detect MeSH-related concepts for indexing; 2) Search-based labeling, which builds on MetaMap-based approach and further leverages information retrieval techniques for finding similar articles whose existing annotations are used for MeSH suggestion; 3) LLDA-based labeling, which further trains a multi-label classifier based on MeSH ontology for MeSH candidate list pruning. The evaluation on the BioASQ challenge data presents promising results and produces interesting findings that may benefit future exploration.
The rest of the paper proceeds as follows: Section 2 highlights the related work. Section 3 describes the data and the task. Then, Section 4 elaborates our methods and Section 5 presents and discusses the evaluation results. Finally, Section 6 summarizes our work and points out future research directions. 2
There are many existing works related to MeSH-based MEDLINE indexing. We
will only highlight a few that are most relevant to our approaches in this section.
The most well-known system for MeSH indexing is the Medical Text Indexer
(MTI) developed at NLM [
in the final stage. Our search-based systems differ from MTI in that we used
MetaMap and information retrieval techniques in a sequential way.
Jimeno-Yepes et al. [
The training set provided by BioASQ challenge contains over 10 million journal articles, each of which consists of the title, abstract, PubMed identifier (PMID), and gold standard MeSH labels that are manually annotated by experts. BioASQ releases 18 test sets of different sizes (ranging from hundreds to tens of thousands documents) over 18 week. Each set consists of new journal articles (<title, abstract, PMID> triples) that have not been annotated or indexed into the PubMed database. The task is to develop systems that can automatically suggest MeSH terms to the unlabeled articles.
We remove duplicated articles that have same PMID in the training set and obtain a pool of 10,699,707 articles with unique PMID5. Furthermore, we randomly sample 4,000 articles from this pool for system training and testing respectively, as shown in Table 1. The reason for creating our own test set instead of using the official ones are two-fold: 1) the gold standard annotations for the BioASQ official test sets are not available to the participants by the time of writing this paper, however, we want to give detailed analysis of our system and present comparable results on both training and testing sets in this paper; 2) we keep adding new features into our system over the whole competition time span, and the official evaluation results do not always reflect the latest development progress of our systems. 5 Note that we will not distinguish between the singular and plural forms of acronyms (such as CUI and DUI) in this paper, i.e., PMID can either stands for PubMed identifier or identifiers depending on the context. Data # of articles Purpose TRN-0 10,691,707 Training data from BioASQ TRN-1 10,687,707 Subset of TN-0 used for finding similar articles to the target article TRN-2 2,000 Subset of TN-0 used for optimizing system parameters TET 2,000 Subset of TN-0 used for evaluation 4
4.1
MetaMap-based Labeling Concept Detection We run MetaMap against an article while restricting its resource to MeSH (i.e., MeSH ontology). We obtain and store the following information: 1) Concepts (denoted as K) which are phrases or terms that map to UMLS CUI; 2) The list L of MetaMap generated CUI candidates c with confidence scores Sc for each K; 3) The negation information for each K. Concept Weighting We first select non-negated CUI whose confidence scores are above the threshold h. Then, we merge and rank these selected c by aggregating their weighted confidence scores. Here we use superscripts T and A to denote title and abstract respectively. The final ranking score of a specific c looks like: score(c) = α
ScL + β
ScL, L∈T
L∈A where α and β are the weights assigned to c in abstract and title respectively, L is the candidate list for each detected concept K, ScL is the confidence score of c in list L. If L does not contain c, ScL will be zero.
In particular, we fix β to 1.0. However, we vary α (i.e., the weights of cT ) to explore the optimal value of α. We use Equation 1 to rank c and select the topranked m ones. Finally, we convert the selected c to MeSH Descriptor Unique Identifiers (DUI).
The above method has three free parameters, i.e., h, α, and m. We set their values by exploring the parameter space as will be described in Section 5.1. where wi is the weight associated with a matched query term qi, and f (qi, D) is the query term matching function defined as: score(Q, D) =
wif (qi, D), qi∈Q f (qi, D) = log tfqi,D + μ tfqi,C
|C| , |D| + μ
In this section, we describe another approach for MeSH suggestion which is based on information retrieval techniques. As aforementioned, our approach starts by finding related articles to the target article, and then leverages their existing annotations to suggest MeSH candidates for the target article.
We use the open-source search engine Indri6 [
Retrieval Model
Our retrieval model computes the relevance score of a document based on the
following function:
(1)
(2)
(3)
where qi is the ith query term used for text matching. Note that qi can be
either a single word or a phrase. |D| and |C| are the document and collection
lengths in words respectively, tfqi,D and tfqi,C are the document and collection
term frequencies of qi respectively, and μ is the Dirichlet smoothing parameter.
Smoothing is a common technique for estimating the probability of unseen words
in the documents [
The above matching function assigns a score to each match of a query term q, and Equation 2 aggregates the scores based on weight w to obtain the final document relevance score. We implement this retrieval model in Indri by formulating queries that look like: #weight(w0 q0 w1 q1 ...wi qi...).
Query Formulation Our next step is to formulate a query Q that can be representative of the content of the article. In this section, we will describe how we effectively generate query terms q as well as their weights w for our ranking function shown by Equation 2. Term Query (TQ) The first type of queries is based on single words/terms in the article, i.e., terms in a term query are all single-word expressions. In particular, we formulate Q based on words occurring in the concepts detected by MetaMap from both title and abstract, i.e., query terms q come from words in KT and KA. Similar to what we have described in Section 4.1, we assign equal weight 1.0 to all qA (i.e., query terms from KA), but use a varying weight γ for all qT . A term query in Indri looks like: #weight(2.0 examination 2.0 cow 2.0 ultrasonographic 3.0 navel 3.0 urachal 3.0 extra-abdominal 2.0 pathologic 2.0 abscess) Phrase Query (PQ) The second type of queries are from KT and KA directly, i.e., we use concepts (usually phrases) as query terms qi. Again, we assign equal weight 1.0 to all qA (i.e., KA), but use a varying weight γ for all qT (i.e., KT ). The following shows an Indri phrase query example: #weight(3.5 #uw2(hiv-1 infection) 4.5 #uw2(differential susceptibility) 2.0 #uw2(actin dynamics) 2.0 actin 4.5 #uw2(cortical actin) 4.5 #uw3(naive t cells) 2.5 dichotomy 3.5 #uw2(human memory) 3.5 #uw3(chemotactic actin activity) 2.0 cd45ro) “#uwN(t1 t2)” means words t1 and t2 can be in any order within a text window of N words, and thus it takes possible variants of a phrase into consideration. Long Query (LQ) The term query considers single words only and ignores the term proximity information in concepts. Thus, it may hurt retrieval precision. On the other hand, the phrase query poses “stricter” matching criteria, i.e., if a relevant document does not have an exact match for a concept phrase K (e.g., for “#uw2(hearing loss)” to match “loss of hearing”), it will not get any credit by Equation 3. Therefore, we formulate a long query that consists of qi from both TQ and PQ, i.e., both single word query terms and phrase query terms. For the above three types of queries (i.e., TQ, PQ, and LQ), to prevent Q from being too long (computationally expensive when retrieving against a large database) we remove q that occur only once in the the abstract and title combined unless all the terms occur only once (which is a very rare case). Result Aggregation For each target article, we formulate query Q and rank documents based on Equations 2 and 3. Then, we take the top-ranked k documents, weight their existing MeSH annotations (i.e., DUI) by their individual relevance scores shown in Equation 2, and aggregate the wegiths for each DUI. Finally, we select the top-ranked m DUI as MeSH annotations for the target article.
In our Search-based Labeling method, we will also allow three free parameters: μ (the Dirichlet parameter in Equation 3), k (the number of top-ranked documents used for DUI aggregation) and m (the number of DUI). We will discuss how to set these parameter in Section 5.1. 4.3
LLDA-based Labeling
The MeSH indexing can also be cast as a multi-labeled classification task.
Therefore, the labeled latent Dirichlet allocation (LLDA) [
In LDA, each document may be viewed as a mixture of various topics, and the topic distribution has a Dirichlet prior. As an extension of LDA, LLDA further incorporates observed label information, and thus can generate topics that predict labels. Therefore, we train an LLDA model with a subset (∼15%) of set of TRN-0 (see Table1) and use the existing MeSH annotations as labels. However, the MeSH ontology contains too many labels (over 25,000 descriptors) for our LLDA to handle. Therefore, we only use MeSH terms at the category level (i.e., children of the root) to form our label set. MeSH annotations of articles are all converted to their corresponding ancestors in this category-level set. Given a target article, our LLDA will predict its category level labels which will be further used to filter irrelevant labels assigned by previous MetaMap-based or search-based systems. Our goal is to remove false positives and improve precision. 5
BioASQ evaluates the MeSH annotations by two different groups of metrics, i.e., flat measures and hierarchical measures, among which Micro F-measure (MiF) and Lowest Common Ancestor F-measure (LCA-F) are the primary evaluation metrics respectively. Thus, we will report Precision, Recall, F-measure for both Microaveraging and LCA measures, i.e., (MiP, MiR, MiF) and (LCA-P, LCA-R, LCA-F).
We have five systems, namely the MetaMap-based system (MM), Search-based systems (TQ, PQ, and LQ), and the LLDA-based system (LLDA). Note that for convenience in the rest of paper we will refer to each system by their short names given in parentheses. 5.1
Parameter Exploration As mentioned in Section 3, we use set TRN-2 to train system parameters. In this section, we show how each free parameter affects performance. MetaMap-based Labeling System MM has three free parameters, i.e., h (title concept weight), α (confidence score threshold for CUI candidates), and m (number of DUI in the final suggested list). To get the best setting for MM, we explore the range (400, 1000, 100) for h, (0, 5.0, 0.5) for α, and (8, 41, 4) for m, and try all different value combinations. Note that the third element is the range is the step size.
In particular, the performance curve in Figure 2(a), where we vary the weight of title concepts, shows that we should assign higher weights to the title concepts. This is expected because the title of an article usually contains the most representative information and the concepts in title are very likely to associate with MeSH annotations.
In Figure 2(b), as we lower the confidence score threshold for MetaMap CUI candidates from 1000 to 700, the precision declines while the recall improves. However, the precision bounces back when h is below 700, and the best performance for MiF, MiP, and MiR all appears at 600.
In Figure 2(c), the precision decreases and the recall increases, both monotonically, as we increase m, the number of DUI for annotating an article. This is also expected because DUI ranked lower down the list are less likely to be correct annotations, and consequently hurt the precision but improve the recall. Search-based Labeling Now we explore the parameter setting for search-based systems, which also have three free parameters: μ (the Dirichlet parameter in Equation 3), k (the number of top-ranked documents used for DUI aggregation) and m (the number of DUI). In particular, we will train system TQ and use it as a reference for setting corresponding parameters in PQ and LQ.
In Figure 2(d), the performance degrades as we increase μ (i.e., more smoothing with the collection-level statistic). This may be because our search-based labeling uses the top-ranked documents for MeSH suggestion and it desires a document set that has a high precision, and on the other hand, less smoothing makes sure that the relevant information remain highly concentrated in these documents which consequently appear among the top of the rank list.
In Figure 2(e), as we increase the number of top-ranked documents the performance peaks early at k = 20 and declines after that point, which is expected because our search-based labeling desires only a few highly relevant documents that can provide a more reliable set of MeSH candidates.
Figure 2(f) looks very similar to Figure 2(c) in which the system tries to strike a balance between precision and recall by varying m. However, method TQ only needs top 10 candidates to achieve the best MiF, as opposed to top 12 in MetaMap-based labeling, indicating that our search-based method is more precision-focused.
Due to the high similarity among search-based systems, we will simply use the best parameter setting of TQ (μ = 125, k = 20, and m = 12) for systems PQ and LQ in our testing stage which is presented next. 0.28 0.27 System TQ, as the simplest among search-based systems, achieves the best performance. Though a direct comparison between TQ and the top-performing BioASQ challenge systems7 is impossible at this stage (since the gold standards have not been released and we have have not submitted runs from this best system), TQ presents quite promising results. However, systems PQ and LQ are doing worse than TQ. The reason might be that the simple term frequency based phrase weighting strategy could not well distinguish important concepts from unimportant ones, and consequently hurts the precision.
In System LLDA, we use the predicted category labels to prune the annotation list from system TQ. We start with a long candidate list by setting m to 20, and then prune this list with LLDA. Table 2(b) shows that LLDA does not produce positive results. This might be because the category level MeSH terms are broad concepts that are not discriminative enough to distinguish one from another. 6
In this paper, we proposed three approaches for automatic MeSH term suggestion: 1) MetaMap-based labeling, which relies on the MetaMap tool to detect MeSH-related concepts for indexing; 2) Search-based labeling, which builds upon MetaMap-based approach and further leverages information retrieval techniques for finding similar articles with existing annotations and uses them for MeSH suggestion; 3) LLDA-based labeling, which further builds on Search-based labeling and trains a multi-label classifier based on MeSH ontology for MeSH candidate list pruning.
Our evaluation on the BioASQ challenge data showed promising results for the Search-based labeling. In addition, we explored the impact of different system parameters (e.g., the weight for title concepts, CUI confidence scores, Dirichlet prior, number of top-ranked documents, etc.) on the system performance. We also proposed a new multi-label classification system based on LLDA for MeSH candidate list pruning. We believe the research findings presented in this paper would be useful for designing similar systems for biomedical semantic indexing. For future work, we plan to explore better concept weighting strategies (e.g., by incorporating corpus-level statistics or using information from external sources) for systems PQ and LQ. As for the LLDA-based labeling, we will extend LLDA model by leveraging hierarchical information in MeSH ontology. In addition, we plan to compare our approaches with existing methodologies and carry out a thorough error analysis to look for aspects that we can further improve.
The work was supported by ABI: 0845523 from United States National Science Foundation, R01LM009959 from United States National Institute of Health. The challenge is organized in the context of the BioASQ project. BioASQ is a Support Action funded under the European Commission FP7 ICT Work Programme, within the Intelligent Information Management objective (ICT- 2011.4.4(d)).