Every year the number of journals and the number of articles to be indexed grows at the U.S. National Library of Medicine (NLM) causing an ever increasing demand on the highly qualified, but, relatively small, dedicated staff of indexers. We present a methodology for identifying MeSH (Medical Subject Headings) Publication Types for assisting the indexers in the categorization of these MEDLINE citations. Publication Types are used by the indexer to describe the type or genre of an article instead of what the article is about, making this a different kind of text categorization problem from identifying MeSH Descriptors. Our goal is to apply a machine learning approach to recommending Publication Types which will save indexers time by providing a precise list of possible Publication Types for each article. Our experiments involved several different machine learning methods to provide Publication Type recommendations which were then evaluated against the gold standard of human indexing. Our results show that machine learning in most cases adds a great deal to the overall performance of recommending Publication Types. Our experiments also show that in some cases, either the full text of the article or feature engineering will be required to accurately produce some Publication Type recommendations.
The MEDLINE®/PubMed® database contains over 21 million citations1. It currently grows at the rate of around 800,000 indexed citations per year covering almost 6,000 international biomedical journals2 in 58 languages. These new citations are manually indexed by a relatively small, dedicated staff of indexers at the U.S. National Library of Medicine (NLM). In this paper, we will use the terms article and citation interchangeably, but they do refer to two distinct entities in the indexing world. Indexers index from the full text of an article, and the results of that effort along with the title and abstract from the article are stored as a citation in the MEDLINE/PubMed data
2 www.nlm.nih.gov/bsd/bsd_key.html base. The indexers use the Medical Subject Headings (MeSH®) 3controlled vocabulary to summarize the central points of full text articles. The 2013 MeSH vocabulary consists of 26,853 MeSH Descriptors4 which are further qualified by a set of 83 Mesh Qualifiers (Subheadings). For example, Aspirin/therapeutic use illustrates the MeSH Descriptor Aspirin being qualified by the MeSH Qualifier therapeutic use showing that the article is not about Aspirin in general, but, more specifically about the therapeutic uses of Aspirin. There are also 214,816 Supplementary Concepts available to the indexer for detailing important chemicals, drugs, or proteins identified in the articles. In addition to summarizing the main points of each article, the indexer is also responsible for other curation tasks such as assigning one or more Publication Types which define the genre of the article.
Publication Types (PTs)5 are a special type of MeSH Heading that are used to indicate what an article is rather than what it is about. There are 61 PTs identified in the four MeSH Publication Characteristics (V) Tree top-level sub-trees that the indexers typically use. These four sub-trees describe a wide range of document types or genres for PTs: Publication Components [V01] (e.g., Architectural Drawings), Publication Formats [V02] (e.g., Eulogies), Study Characteristics [V03] (e.g., Clinical Trial), and Support of Research [V04] (U.S. Government and non-U.S. Government) with some PTs included in multiple sub-trees. Multiple PTs can be assigned to the same article by the indexer.
The ever increasing demand for indexing (502,056 indexed in 2002 to 760,903
indexed in 2012, and with NLM expecting to index over one million articles annually
within a few years) is a growing and burdensome workload in a time of dwindling
resources. NLM created the NLM Indexing Initiative (II) [
In 2004, testing MTI’s PT recommendations showed that MTI was not very good at
the task, as shown in Table 1. MTI has two main methods of summarizing what a
citation is about: MetaMap Indexing (MMI) [
We have studied the use of various machine learning algorithms testing their ability to accurately recommend several different types of PTs for MEDLINE citations. We have selected the following ten PTs to see if we could provide reliable recommendations. Case Reports: Clinical presentations that eventually lead to a diagnosis. Clinical Trial: Work that is the report of a pre-planned clinical study. Congresses: Published records of the papers delivered at or issued on the occasion of individual congresses, symposia, and meetings. Controlled Clinical Trial (CCT): Work consisting of a clinical trial involving one or more test treatments and at least one control treatment. Editorial: Work consisting of a statement of the opinions, beliefs, and policy of the editor or publisher of a journal. English Abstract: English Abstracts of foreign articles. In Vitro: Studies using excised tissues. Meta-Analysis: Work consisting of studies using a quantitative method of combining the results of independent studies. Randomized Controlled Trial (RCT): Similar to Controlled Clinical Trial, but requires that the treatments to be administered are selected by a random process. Review: An article or book published after examination of published material on a subject.
These ten PTs were selected because they represent some of the most frequently used PTs and provide a good cross category sample of the four Publication Characteristics MeSH tree sub-trees for PTs. We also limited our set to 10 PTs to facilitate training and evaluation.
As mentioned before, changes in the indexing policy can have a dramatic effect on how articles are indexed and can create inconsistencies in a large training corpus if special care is not taken. To reduce the chance of this, we have focused on the last three full indexing years using the 2012 MEDLINE Baseline. We used the Medline Baseline Repository Query Tool6 to identify a list of PMIDs (PubMed Unique Identifiers) for Date Completed (date indexing was applied to the citation) ranging from January 1, 2009 to December 31, 2011. The Query Tool also allowed us to randomly divide the list of PMIDs into Training (2/3) and Testing (1/3) sets. We ended up with 1,784,061 randomly selected PMIDs for Training and 878,718 for Testing. Once we had the two lists of PMIDs, we extracted the actual citations from the 2012 MEDLINE Baseline in XML format for use with our MTI ML machine learning package7. The MTI ML package was developed as part of the Indexing Initiative effort to provide machine learning algorithms optimized for large text categorization tasks and capable of combining several text categorization solutions. It is available subject to the MetaMap Terms and Conditions8.
Certain types of articles require special indexing. For example, a Comment On article, which is an article commenting on a different article, is indexed by simply using the indexing from the originating article. For a Review type of article, the indexer uses fewer MeSH Headings that tend to be more general in nature than they would use for a non-Review article. For these reasons, when we assembled the final data set, we also filtered out the articles requiring special handling to create as clean a data set as possible. Specifically, we removed the following types9 of articles from our data sets: OLDMEDLINE, PubMed-not-MEDLINE, articles with no indexing, CommentOn, RetractionOf, PartialRetractionOf, UpdateIn, RepublishedIn, ErratumFor,
7 http://ii.nlm.nih.gov/MTI_ML/index.shtml 8 http://metamap.nlm.nih.gov/MMTnCs.shtml 9 http://www.nlm.nih.gov/bsd/licensee/elements_alphabetical.html and ReprintOf. This left us with 1,321,512 articles for Training and 651,617 articles for Testing. The data sets used for these experiments are available from our Indexing Initiative Data Sets and Test Collections web page10.
The task of assigning PTs to a MEDLINE citation can be seen as a text categorization
task [
SVM has been shown to perform well on text categorization tasks [
One of the algorithms that we have extensively used is AdaBoostM1 (Ada) using an
implementation of decision trees based on C4.5 as the base learning algorithm. In
previous work, Ada had performed well on the Check Tags set [
For each machine learning method, we trained with up to four different feature variations. In all the cases, we considered only Boolean features, either the feature appears in the citation or not: 1. Base method, which includes the text from the Title and Abstract fields. The text has been tokenized, lowercased, and no stemming was applied. 2. Base method plus added text features (-F). For the added text features, we also include the following fields to the default Title and Abstract fields for training: Journal Unique Identifier, Author Affiliations, Author Names, and Grant Agencies. Some of the features rely on either the authors or the institutions to be working on the same type of publications, which might change after some time. The plan is to retrain the learning algorithms to avoid any concept drift.
Due to time constraints, AdaBoostM1 was only trained using the first two variations.
We used five different machine learning methods: Modified Huber Loss (Mhl), Hinge
Loss (Sgd), Naïve Bayes (NB), Logistic Regression (LR), and AdaBostM1 (Ada).
So, in the table under methods, “Mhl-BF” means Modified Huber Loss using bigrams
and added text features. We have also highlighted the four PTs (CO, EA, IV, and
MA) where we have baseline results from early MTI performance. Even though not
directly comparable, the difference in performance is quite significant.
For the four PTs that we have baseline performance information, we can see three
have a dramatic improvement with machine learning: Congresses improves from
0.3397 to 0.7113 (+109%), English Abstract improves from 0.0010 to 0.8359
(+835%), and Meta-Analysis improves from 0.2674 to 0.7742 (+190%).
Interestingly, In Vitro actually has a decrease in performance from 0.1679 to 0.1610 (-4%).
Mhl-BF and LR-BF have the best performance from the evaluated methods. These
two classifiers have already shown better performance compared to other algorithms
in existing work on MeSH indexing [
Not surprisingly with machine learning, there is no clear winning method that works
best for all of the Publication Types, echoing the findings for MeSH indexing
[
The Naïve Bayes method was far behind all of the other methods. This effect is more dramatic when the ratio of positives is smaller compared to the number of negatives. This has been explained already by Rennie et al. [Error! Reference source not found.] and it is due to the imbalance between the classes for which the Naïve Bayes classifier favors the majority class. In addition, this effect is more dramatic with a larger set of dependent features, in which the decision boundary is pushed by the related features favoring the majority class even more.
Case Reports, Congresses, English Abstract, Meta-Analysis, Randomized Controlled Trial, and Review all have F1 measures above 0.700 making them promising candidates for future integration into the indexing process. The remaining PTs Clinical Trial, Controlled Clinical Trial, Editorial, and In Vitro all have F1 measures too low for consideration at this time but provide the kernel for further research into improving their performance.
The overall results are promising enough to warrant expanding the experiments to include more PTs to see how they will perform.
If we focus on the 480,631 citations in the 2013 MEDLINE Baseline with a 2012 Publication Date, we can see that several of our high performing PTs were also some of the most frequently used PTs. Review (46,808) is fourth, Case Reports (27,662) fifth, English Abstract (14,208) tenth, and Randomized Controlled Trial (11,408) twelfth. By providing accurate Publication Type recommendations to the indexers, we will help make their jobs easier and more efficient.
Two of the PTs intrigued us enough to warrant a deeper study for very different reasons.
English Abstract performed very well (0.8359) in our experiments, but, we could not understand why it did not reach 1.0000. The rule for identifying whether an article is actually an English Abstract is very clear, more so than most of the PTs. If an article has a title in brackets (meaning it was translated into English) and contains an abstract, it should receive the English Abstract Publication Type. What we found in talking with an indexer is that English Abstract is actually not added by the indexer at all. This rule was straightforward enough that there is a program in place to automatically assign this Publication Type to articles before the indexing is released to the MEDLINE/PubMed database. During our false positives error analysis we found that the majority of cases met the definition of English Abstract, but, simply did not have the Publication Type assigned and this is very likely the cause of not meeting our goal of 1.0000.
In Vitro on the other hand actually performed worse than our MTI baseline and we wanted to try and find out what might be causing this anomaly. In Vitro was designated as a Check Tag when our MTI baseline measure was taken and changed to being a Publication Type shortly thereafter. As a Check Tag, indexers would have used In Vitro much differently than as a Publication Type since Check Tags are based on the main topics found in the article and PTs describe the type or genre of the article. This may account for some of the differences in performance, but, there had to be additional reasons for such a low F1 measure (0.1610) for In Vitro. We only used the last three years of MEDLINE in our experiments, so this time period would only include In Vitro as a Publication Type. So, we should not be confusing the machine learning algorithms by providing them with contradictory data. What we found in our error analysis was that in almost all of the false negatives that we manually reviewed, the information for designating the article as In Vitro was located in the full text of the article, usually in the Methods section, where the authors describe how they performed their research. This fact alone explains the low performance for In Vitro and highlights one of the challenges we mentioned earlier (full text versus only using title and abstract) to successfully recommend Publication Types. 5
We have evaluated the automatic assignment of PTs to MEDLINE articles based on machine learning, which extends our previous machine learning efforts using MTI. We find that for the majority (6 of 10) of PTs the performance is quite good with F1 measures above 0.700, while further work is required for the rest of them. The results also show that in addition to the title and abstract text, further information provided from fields in the MEDLINE article result in improved performance. The discussion section shows that feature engineering might provide improved performance, for instance, in the English Abstract case.
Future work will involve expanding the experiments to include most of the remaining frequently used PTs to see if we can identify the set of PTs that perform the best and that would provide the most assistance to the indexers. We will also be exploring the use of openly available full text from PubMed Central14 to see if the full text would benefit In Vitro as well as other poorly performing PTs.
NICTA is funded by the Australian Government as represented by the Department of Broadband, Communications and the Digital Economy and the Australian Research Council through the ICT Centre of Excellence program. This work was also partly supported by the Intramural Research Program of the NIH, National Library of Medicine. The authors would also like to thank Preeti Kochar a senior indexer at the U.S. National Library of Medicine for her valuable insights into how the Publication Types work from an indexer’s perspective. 14 http://www.ncbi.nlm.nih.gov/pmc/ 12. Zhang, T, Solving large scale linear prediction problems using stochastic gradient descent algorithms. In Proceedings of the Twenty-First International Conference on Machine Learning. ACM, 2004. 13. Yeganova L, Comeau DC, Kim W, Wilbur WJ. Text mining techniques for leveraging positively labeled data. In Proceedings of BioNLP 2011 Workshop (pp. 155-163). Association for Computational Linguistics. 14. Rennie J.D., Shi Rennie J.D., Shih L., Teevan J., Kerger DR. (2003) Tackling the poor assumptions of naive bayes text classifiers. ICML. Vol. 3. 2003.