Comparing Word Embeddings for Document Screening based on Active Learning

Abstract

Document screening is a fundamental task within Evidencebased Medicine (EBM), a practice that provides scienti c evidence to support medical decisions. Several approaches are attempting to reduce the workload of physicians who need to screen and label hundreds or thousands of documents in order to answer speci c clinical questions. Previous works have attempted to semi-automate document screening, reporting promising results, but their evaluation is conducted using small datasets, which hinders generalization. Moreover, some recent works have used recently introduced neural language models, but no previous work have compared, for this task, the performance of di erent language models based on neural word embeddings, which have reported good results in the latest years for several NLP tasks. In this work, we evaluate the performance of two popular neural word embeddings (Word2vec and GloVe) in an active learning-based setting for document screening in EBM, with the goal of reducing the number of documents that physicians need to label in order to answer clinical questions. We evaluate these methods in a small public dataset (HealthCLEF 2017) as well as a larger one (Epistemonikos). Our experiments indicate that Word2vec have less variance and better general performance than GloVe when using active learning strategies based on uncertainty sampling.

Keywords

• active learning
• evidence based medicine
• document screen- ing
• word embeddings

-

Evidence-based Medicine (EBM) is a practice that provides scienti c evidence to support medical decisions. This evidence nowadays is obtained from biomedical journals, usually accessible through the portal PubMed1, a search engine which provides free access to abstracts of biomedical research articles as well as to the MEDLINE database. An existing problem is to nd relevant documents within a massive volume of documents, given a clinical question or a query. As a consequence of this, the time required for the search and screening of articles

1 https://www.ncbi.nlm.nih.gov/pubmed/

related to clinical questions about medical problems can take long and sometimes it consumes a large part of a physician's workday [ 15, 6 ]. When people conduct this repetitive task, there is a good chance of overlooking important articles, which can have a negative impact on decisions such as the patient's treatment [ 12 ]. Moreover, the publication of medical papers has grown exponentially the last decade. Since 2005, PubMed has indexed more than 1 million articles per year, which means that the process of searching and manual screening of medical evidence will become increasingly more di cult for physicians without the support of information retrieval and machine learning algorithms. For this reason, some systems have emerged to support experts in the collection of evidence such as Embase2, DARE3 and Epistemonikos4. In this article, we work with data from Epistemonikos, which helps expert physicians to review and validate scienti c evidence grouped by medical questions to facilitate its subsequent search. Our goal is to improve the e ciency and e cacy of document screening in the practice of EBM. In other words, we aim at reducing the e ort made by physicians at screening documents to nd the evidence needed to support the answers of a medical question. We use an active learning approach, experimenting with a large dataset of medical questions, unlike previous works which use very small datasets, some of them very recent [ 13 ]. In this short paper, we contribute by: i ) Experimenting in both a large dataset (Epistemonikos, 987) and a small dataset (CLEF, 50), showing evidence of generalization of our approaches, and ii ) comparing the performance of documents represented with two state-of-the-art neural word embeddings (Word2vec [ 14 ] and GloVe [ 18 ]) as well as traditional relevance feedback [ 5 ]. 2

Related Work

The task of nding relevant documents related to a medical question through citation screening has been studied and it is known as the total recall problem: given a medical topic or question, nd all the documents that are relevant about a particular topic. Recently, the CLEF task 2 [ 11 ] is a challenge that calls for solving the problem of prioritizing which documents to screen to reduce work overload for experts. They provide a public dataset with medical topics and a set of candidate documents; participants have to rank documents by relevance for every speci c medical subject in the minimum of iterations to make more e cient the document screening process. In the literature, the approaches to solving this problem are based on two general lines: information retrieval and machine learning methods.

In the information retrieval area, there have been many attempts to solve the problem using techniques such as relevance feedback [ 5 ], query expansion [ 13 ], ranking and inference based on external knowledge [ 8 ]. However, they do not 2 https://www.elsevier.com/solutions/embase-biomedical-research 3 https://www.crd.york.ac.uk/CRDWeb/ 4 https://www.epistemonikos.org/en ensure a level of recall necessary to capture all the evidence related to a medical question.

From the machine learning community, the approach is to automate or semiautomate the screening process or review of medical articles that were previously selected as relevant to a medical question by learning the pattern of physicians conducting a document survey. There have been e orts to solve this problem by using automatic classi cation [ 2, 3, 1, 16, 21 ]. Where they compared classi ers such as Naive Bayes, K-NN, and SVM, using di erent ways to represent text, such as word embeddings and bag-of-clinical terms from titles and abstracts. There is also literature that has used active learning [ 9, 7, 22, 15 ] for medical topic detection and clinical text classi cation. Moreover, a few of deep learning models have been proposed for the classi cation of relevant evidence and categorization of documents in medical questions [ 4, 10 ]. Generally, the majority of work done has used datasets of up to 50 medical topics/questions and 200,000 documents, and in this case, we work with a dataset close to 1; 000 medical questions and 370; 000 potential documents, allowing models to generalize and obtain better e cacy results compared to the state of the art. In addition, unlike previous work, we compare two neural word embedding models for document representation (Word2vec [ 14 ] and GloVe [ 18 ]) in order to asses their performance for the biomedical document screening task. 3

Proposed Solution

The process of nding documents that answer a clinical question requires rst retrieving a set of candidate documents. Then, physicians perform the document screening where they verify that abstracts and titles of each document are related to the medical question, this particular process may involve much time and cognitive e ort from experts.

Problem formulation: Given a medical question q and a set of candidate documents C = fc1; c2; :::cng we need to ask an oracle (physician) O to label these documents as relevant or not relevant to q. We want to avoid asking the labeling of every document, so we select an informative sample to be labeled by the expert. With these labels, we train a predictive model M . It might be necessary to ask for labels in many iterations in order to re ne the model, ending up with several models M0; M1; :::; Mk.

In our case, we use an active learning (AL) approach [ 20 ]. Using an AL strategy A (e.g., uncertainty sampling, query-by-committee, etc.), we sample a set of unlabeled documents X from C, in order to ask the oracle O to label them. With the labeled items, we then train a machine learning model Mi(X; Y ) with the new observations X C with labels Y (binary, Y = 1 means relevant document and Y = 0 means not relevant) given by O. Then, we use the trained model Mi to predict relevance labels for unobserved documents, and using the active learning strategy A we select new items to be labeled by O in order to create an updated version of the model Mi+1. In each iteration, we evaluate the model (precision, recall, ), and we can stop until a xed number of iterations or after the model converges.

To address the problem above, we developed a system, where we start with a small proportion of labeled documents as relevant or not relevant for each medical question to train a rst version of the machine learning model Mi. Then, using the active learning strategy we chose instances to be labeled by a physician based on the title and abstract text features represented internally as word embeddings (GloVe and Word2vec). After the physician adds the labels, they are used to train a machine learning model Mi+1 to predict the relevance of new unlabeled documents and thus begin a new iteration.

The performance of our approach rst depends on the machine learning algorithm chosen and second, on the active learning strategy that chooses unlabeled examples to create a labeled dataset as input for supervised learning algorithms. The strategies used in this experiment are uncertainty sampling and random sampling, given their lower complexity compared to others such as error-based, gradient-based and variable reduction [ 20 ]. The machine learning algorithms that are considered to be trained with new labeled examples are random forests, logistic regression, and neural networks.

With respect to the active learning sampling strategies, random sampling: chooses random documents to train the machine learning model and it is usually used as a comparison baseline against other approaches. On the other side, uncertainty sampling: looks for records which have higher label prediction uncertainty, making them potentially more informative for collecting their actual labels and then training or updating a model. 4

Experiments

Dataset. For the experiments, we used two datasets: CLEF5 and Epistemonikos. Both of them have a similar distribution of documents per question, where the majority of medical questions contain an approximate number of 200 relevant documents. On the one hand, CLEF dataset contains only 50 medical questions and 200,000 documents related to them that were crawled from PubMed using each document id. On the other hand, the Epistemonikos Evidence Synthesis Project is a collaborative initiative established in 2012 with the objective of collecting, organizing and comparing all relevant evidence for health decisionmaking, through a multilingual platform. This dataset is composed of 987 medical questions and 372,829 potential documents. In both datasets each medical question is associated to a Systematic Review (hereinafter, SR), which is a type of article that collects and synthesizes the most relevant primary studies and trials related to a question. The information of documents from both datasets consists of the title, abstract, author, year and the label if it is relevant (or not) to the question or medical subject. In the case of Epistemonikos data, the labels were previously curated by senior medical students, in which they had to select papers related to a set of medical questions. Document representation: for each

5 https://sites.google.com/site/clefehealth2017/task-2

document we lower case the concatenation of title and abstract, then remove stop words, and we use GloVe [ 18 ] and Word2vec [ 14 ] to obtain an embedding representation of 300 dimensions of each word. Finally, using average pooling we obtain a vector for each document. 4.1

O ine Active Learning Setup We experimented doing a simulation of the active learning labeling process of documents for medical questions. As each medical question has a di erent number of relevant documents, we sample documents that are not relevant where the total of relevants corresponds to the 5%, so that the distribution of documents is similar to the CLEF dataset [ 13 ]. We ltered out some of the medical questions, keeping those that have more than ve relevant documents and less than 2,000 relevant documents, ending up with 987. We compared the results of applying active learning on the CLEF dataset that contains 50 SR (Systematic Reviews) with Epistemonikos. For each medical question, we hide the document labels and we leave only ve with their respective labels to start building the model and then iterating with active learning to receive feedback from the oracle. For each prediction made by the machine learning model in each iteration, we sorted the results depending on the predicted probability of being relevant for each model, so the evaluation metrics were calculated with the ranked list of potential candidates given by each strategy. The parameters chosen for machine learning algorithms were: for neural networks we used ve hidden layers, ReLu activation function, learning rate of 1e-05, momentum of 0.9, 100 neurons per layer and Adam optimization function. For the random forest, we used 100 estimators. Experiments were programmed in Python3 using libact [ 23 ], scikit-learn [ 17 ], pandas and gensim libraries. Code for these experiments will be published in a github repository after noti cation.

Evaluation metrics. We evaluated our proposed active learning strategies with traditional IR metrics also used by Lee et al. [ 13 ]: precision@k, recall@k and mean average precision (MAP). We report the metrics obtained after ten iterations, with ten documents labeled per iteration. 5

Results and Discussion

Average Precision (MAP) performance measured in Epistemonikos and CLEF datasets using active learning strategies (US: uncertainty sampling, RS: random sampling) using a batch of 10 documents per feedback iteration for Word2vec and GloVe representation.

Dataset Epistemonikos

987 SRs GloVe 300 dim Epistemonikos

987 SRs Word2vec 300 dim

CLEF 50 SRs GloVe 300 dim

CLEF 50 SRs Word2vec 300 dim

AL-Model

MAP US-NN US-RF US-LR RS-LR US-NN US-RF US-LR RS-LR US-NN US-RF US-LR RS-LR US-NN US-RF US-LR RS-LR

6 Conclusion and Future Work

In this article we supported results from previous studies in terms of showing that active learning with an uncertainty sampling strategy yields good results for the task of biomedical document screening. Moreover, we contribute by comparing two popular word embeddings to represent documents: Word2vec and GloVe. The best results were obtained using Word2vec document representation and random forests as the learning algorithm. GloVe document representation also yields competitive results, but it seems more sensitive two the classi cation model used: it performs well with random forests but shows poor performance with neural networks and logistic regression. Moreover, our experiments indicate that these results are consistent in both the small public dataset of HealthCLEF and the larger dataset of Epistemonikos, giving evidence of generalization. For future work, we will try other machine learning models, active learning strategies and evaluate the results using CLEF metrics [ 11 ]. We will also test other paradigms for more scalable learning, such as weak supervision. With respect, to embeddings, we will test di erent values of sensitive parameters, as mentioned by Roy et al. [ 19 ]. Finally, we will conduct a user study with actual physicians in order to evaluate online the performance of our approach.

7 Acknowledgements

We acknowledge Epistemonikos Foundation, the Chilean research agency Conicyt, Fondecyt grant 1191791 and the Millenium Institute IMFD.

References

1. Adeva , J.G. , Atxa , J.P. , Carrillo , M.U. , Zengotitabengoa , E.A. : Automatic text classi cation to support systematic reviews in medicine . Expert Systems with Applications 41 ( 4 ), 1498 { 1508 ( 2014 )

2. Bekhuis , T. , Tseytlin , E., Mitchell, K.J. , Demner-Fushman , D. : Feature engineering and a proposed decision-support system for systematic reviewers of medical evidence . PloS one 9 ( 1 ), e86277 ( 2014 )

3. Choi , S. , Ryu , B. , Yoo , S. , Choi , J.: Combining relevancy and methodological quality into a single ranking for evidence-based medicine . Information Sciences 214 , 76 { 90 ( 2012 )

4. Del Fiol , G. , Michelson , M. , Iorio , A. , Cotoi , C. , Haynes , R.B.: A deep learning method to automatically identify reports of scienti cally rigorous clinical research from the biomedical literature: Comparative analytic study . J Med Internet Res 20 ( 6 ) ( Jun 2018 )

5. Donoso-Guzman , I. , Parra , D.: An interactive relevance feedback interface for evidence-based health care . In: 23rd International Conference on Intelligent User Interfaces . pp. 103 { 114 . ACM ( 2018 )

6. Elliott , J.H. , Turner , T. , Clavisi , O. , Thomas , J. , Higgins , J.P. , Mavergames , C. , Gruen , R.L. : Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap . PLoS medicine 11(2) , e1001603 ( 2014 )

7. Figueroa , R.L. , Zeng-Treitler , Q. , Ngo , L.H. , Goryachev , S. , Wiechmann , E.P. : Active learning for clinical text classi cation: is it better than random sampling ? Journal of the American Medical Informatics Association 19 ( 5 ), 809 { 816 ( 2012 )

8. Goodwin , T.R. , Harabagiu , S.M. : Knowledge representations and inference techniques for medical question answering . ACM Transactions on Intelligent Systems and Technology (TIST) 9 ( 2 ), 14 ( 2018 )

9. Hashimoto , K. , Kontonatsios , G. , Miwa , M. , Ananiadou , S. : Topic detection using paragraph vectors to support active learning in systematic reviews . Journal of biomedical informatics 62 , 59 { 65 ( 2016 )

10. Hughes , M. , Li , I. , Kotoulas , S. , Suzumura , T. : Medical text classi cation using convolutional neural networks . Stud Health Technol Inform 235 , 246 { 50 ( 2017 )

11. Kanoulas , E. , Li , D. , Azzopardi , L. , Spijker , R.: Clef 2017 technologically assisted reviews in empirical medicine overview . In: CEUR Workshop Proceedings . vol. 1866 , pp. 1 { 29 ( 2017 )

12. Keselman , A. , Smith , C.A. : A classi cation of errors in lay comprehension of medical documents . Journal of biomedical informatics 45(6) , 1151 { 1163 ( 2012 )

13. Lee , G.E. , Sun , A. : Seed-driven document ranking for systematic reviews in evidence-based medicine . In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval . pp. 455 { 464 . ACM ( 2018 )

14. Mikolov , T. , Sutskever , I. , Chen , K. , Corrado , G.S. , Dean , J. : Distributed representations of words and phrases and their compositionality . In: Advances in neural information processing systems . pp. 3111 { 3119 ( 2013 )

15. Miwa , M. , Thomas , J. , OMara-Eves , A. , Ananiadou , S. : Reducing systematic review workload through certainty-based screening . Journal of biomedical informatics 51 , 242 { 253 ( 2014 )

16. Mo , Y. , Kontonatsios , G. , Ananiadou , S. : Supporting systematic reviews using lda-based document representations . Systematic reviews 4(1) , 172 ( 2015 )

17. Pedregosa , F. , Varoquaux , G. , Gramfort , A. , Michel , V. , Thirion , B. , Grisel , O. , Blondel , M. , Prettenhofer , P. , Weiss , R. , Dubourg , V. , et al.: Scikit-learn: Machine learning in python . Journal of machine learning research 12(Oct) , 2825 { 2830 ( 2011 )

18. Pennington , J. , Socher , R. , Manning , C. : Glove: Global vectors for word representation . In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) . pp. 1532 { 1543 ( 2014 )

19. Roy , D. , Ganguly , D. , Bhatia , S. , Bedathur , S. , Mitra , M.: Using word embeddings for information retrieval: How collection and term normalization choices a ect performance . In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management . pp. 1835 { 1838 . ACM ( 2018 )

20. Settles , B. : Active learning . Synthesis Lectures on Arti cial Intelligence and Machine Learning 6 ( 1 ), 1 { 114 ( 2012 )

21. Wallace , B.C. , Small , K. , Brodley , C.E. , Lau , J. , Schmid , C.H. , Bertram , L. , Lill , C.M. , Cohen , J.T. , Trikalinos , T.A. : Toward modernizing the systematic review pipeline in genetics: e cient updating via data mining . Genetics in medicine 14(7) , 663 ( 2012 )

22. Wallace , B.C. , Small , K. , Brodley , C.E. , Trikalinos , T.A. : Active learning for biomedical citation screening . In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining . pp. 173 { 182 . ACM ( 2010 )

23. Yang , Y.Y. , Lee , S.C. , Chung , Y.A. , Wu , T.E. , Chen , S.A. , Lin , H.T.: libact: Poolbased active learning in python ( 2017 )