With the tremendous growth of biomedical literature and data, it's no longer feasible for researchers to manually sift through this information for answering questions on speci c topics. The \Machine Reading of Biomedical Texts about Alzheimer Disease" task of CLEF QA4MRE encouraged the development of systems that can automatically nd answers to questions on Alzheimer disease. To this end, we developed several information retrieval(IR) and semantic web-based strategies. Our best performing strategy used a combination of query processing followed by IR on the background corpus, distributed by the organizers, to nd correct answers. Using our systems, the highest cumulative and individual c@1 scores achieved were 0.47 and 0.66 respectively.
Research in the biomedical domain has seen unprecedented growth in the past few decades. While the majority of biomedical research is still conducted in wet-labs on plant and animal models i.e. in vivo, there has been a steady rise in the use of automation and in-vitro techniques in this area. The result is an ever-increasing volume of data and literature which can no longer be processed manually. Making sense of this insurmountable amount of data and information without computational techniques is practically impossible for life sciences researchers. To this end, various biomedical text mining applications have been developed to ful ll the information need for life sciences researchers. Information retrieval and automated question answering (QA) are essential examples of such applications which have found increasing importance among researchers.
Biomedical information retrieval is a thriving domain. Retrieval systems like PubMed cater to the information need of thousands of users every day. In 2011 more than 1.8 billion PubMed searches were conducted, an increase of 13% from the year before1. While such retrieval systems facilitate the search for documents relevant to users' queries, it does not provide precise answers to questions. In
contrast, QA takes a ne grained approach to information retrieval in providing
precise answers to questions. In essence, it can be viewed as an information
retrieval problem where the task is to nd sections (phrases/sentences/paragraphs)
of an article that are relevant to a question rather than nding the entire
article[
QA in the biomedical domain has two major challenges. First, entities
involved in the question can have synonyms, abbreviations and various sources
of ambiguity which makes the search process challenging. Second, in practical
settings, with the availability of millions of biomedical articles, medical records
and domain-speci c thesauri, answers to questions can vary widely depending
on the domain under discussion [
The \Machine Reading of Biomedical Texts about Alzheimer Disease" task of CLEF QA4MRE aimed at exploring and evaluating systems designed for answering questions about Alzheimer disease. Similar to QA tasks in the biological domain, the goal was to nd precise answers to questions from one or more corpora of biomedical texts on Alzheimer disease. In our implementation of systems to address this problem, we followed various information retrieval-based sentence extraction approaches in nding the most suitable answer to a given question. The rest of the paper is organized as follows: Section 2 reviews some of the related literature in the domains of medical as well as biological QA, Section 3 discusses the details of the task and the datasets, Sections 4 and 5 elaborate on the various strategies used for our submitted and un-submitted runs along with the results, and nally, in Section 6 we outline future work in this direction. 2
Automated QA has undergone tremendous progress in the recent years. QA systems such as IBM's `Watson' gained prominence in popular culture through participation and eventually winning the `Jeopardy! Challenge'2. `Watson' used Apache UIMA's real-time content analytics3 in conjugation with deep natural language processing, information retrieval, machine learning, etc. to provide answers to open domain questions in an extremely e cient way. Application of such advanced systems to QA in specialized domains such as Alzheimer disease would be quite interesting. Other than such domain independent QA systems there has been considerable research on QA for medical and biological domains.
Evidence-based medicine (EBM) is one of the primary motivators for
medical/clinical QA [
Several systems have been developed over the years for clinical QA. Most
systems use some domain speci c knowledge for nding answers to questions.
Following the PICO standards, [
The strategies for QA in the biological domain mimics the practices of
clinical QA (except for the PICO framework). Semantic-based approaches use the
UMLS metathesaurus and other thesauri for query expansion strategies and
shortlisting candidate articles as answers [
Similar to recent research on nding important segments of biomedical text
for document summarization [
The objective of the \Machine Reading of Biomedical Texts about Alzheimer
Disease" task, as the title suggests, is to explore various strategies of machine
reading systems to answer questions pertaining to Alzheimer disease4. In
comparison to other domains, biomedical text has its unique challenges (as
discussed before). Over the past couple of decades various tools and applications
have been designed for so-called micro-tasks on biomedical text, namely,
information extraction [
In the Alzheimer QA task the focus is on reading single documents and identi cation of answers for a set of questions using information implicitly or explicitly stated in the text. Systems developed for this task are required to identify a correct answer from a set of 5 probable answers for a question in a Multiple Choice Question (MCQ) setting. To identify the correct answers systems may use a reference document collection on Alzheimer disease provided by the Lab organizers. It is important to note here that answers to questions pertaining to a particular document can be detected using that document only, while systems may bene t from using the background collection and associated pre-processed information made available to the participants. 3.2
PubMed abstracts A set of 66,222 abstracts relevant to Alzheimer disease was obtained using PubMed search and made available to the participants.
Access full text articles were obtained from PMC in PDF format. 7,512 of these articles were converted into text format using LA-PDFText5.
A smaller set of 1,041 full text articles in HTML and text format from the last three years on Alzheimer disease was also obtained from PMC. Elsevier full text articles A set of 379 full text articles and 103 abstracts were obtained from Elsevier in XML and text format. This set, containing articles referring to 45 core hypothesis in Alzheimer disease, was manually selected by an expert in this area.
Annotated Data The documents of the background collection were annotated
across di erent dimensions using various publicly available tools. The
dependency parser, GDep [
Training Data The training set comprised of a single full-text XML-formatted article along with questions and answers in the MCQ format described above. Correct answers to all the questions were also made available to the participants.
Test Data The test set comprised of 4 full-text documents, each containing 10 MCQ questions in a format similar to the training set. Both training and test documents were processed with the annotation strategy outlined above.
Questions in the training and test set could be classi ed into 3 degrees of difculty, namely, simple (answer present almost verbatim in the article), medium (questions containing lexico-semantic alienations of NEs), complex (reasoning and derivation-based questions). The type of questions spanned across various topic types such as identi cation of experimental evidences, protein-protein interactions, gene synonymy relations or regulatory relations, in the context of Alzheimer disease. 4 4.1
Greek alphabet expansion In the rst step we expand all Greek alphabets into their corresponding English names. For example, `Ab' is converted into `Abeta'.
Dictionary generation In this step we parsed the Elsevier articles to identify
expanded forms of abbreviations (marked by `QUALIFIER Abbreviations' in the
document collection). A dictionary of abbreviations and their corresponding full
forms was created and applied to the questions and answers of the test ocuments.
POS tagging In this step we used the Stanford Log-linear POS tagger [
Sentence splitting We used the GENIA sentence splitter [
Indexing We created several indexes from the test document and the
background collection. The Indri information retrieval tool [
A single index was created by combining the above 3 indexes. This is referred to as CombinedIndex in the remainder of the paper. 4.3
We submitted 7 o cial runs for this task. In this section we highlight the major strategies underlying each run. For all runs except two we followed a two step retrieval approach. Generally, in the rst step we retrieved a set of candidate sentences that may contain the correct answer to a question. In the second step a single correct answer from this pool of candidates is selected using retrieval techniques. For two of the submitted runs we used a slightly di erent strategy. In one strategy we used a hypothesis generation technique using both the questions and answers of QA task for nding the correct answers. In the other strategy we followed a majority voting scheme for selecting the correct answer from a pool of runs.
Run 1 In this run we used the preprocessed questions (i.e. questions processed using the steps outlined in Section 4.1) for retrieving a candidate set of sentences that might contain the correct answer. The retrieval of sentences was done only on the test document for that question using Indri's default language model. For querying we used Indris belief operator (#combine) which, unlike Boolean operators (e.g. AND, OR, etc.) that returns only binary values, weighs each term equally and prioritizes documents (sentences in this case) containing more query terms. Stop words were removed for every retrieval experiment. We limited our retrieval to the top 5 sentences from the TestIndex.
In the second step we followed the same pre-processing steps as before but only on the answers. Again we used Indri's #combine operator to retrieve the correct answer for each question. An answer choice that's also a part of a question was automatically discarded as a candidate. To identify the correct answer to a question we count the number of sentences (out of 5) that are retrieved for each answer. The answer that retrieved the most number of sentences was considered as the correct one. In case of a tie, we did not answer that particular question. Run 2 In this run we followed the steps identical to Run 1 except that we used a tf*idf retreival model instead of Indri's default language model. Run 3 This run is also similar to Run 1 but here we did not limit our retrieval to the top 5 candidate sentences in the rst step. As a result, the retrieval in the second step is based on a larger pool of sentences from which we select the correct answer as the one retrieving the most number of sentences from the candidate pool. Similar to Run 2 we use the tf*idf retrieval model instead of the default language model of Indri. In case of a tie, we skip answering that question. Run 4 In this run, we adopted the same strategy as in Run 3 in the rst step. However in the second step, we selected a correct answer by ranking the retrieved sentences by tf*idf retrieval scores and selecting the answer corresponding to the highest scoring sentence as the correct one.
Run 5 In this run we followed similar strategies as in Run 2, but instead of selecting the correct answer by the count of retrieved documents we used the highest retrieval score for identifying the correct answer.
Run 6 This run followed a considerably di erent strategy compared to all of the previous runs. Here each preprocessed question was combined with all possible answers to that question to form various hypotheses. For example the question \Which technique was used to determine the cellular locations of the CLU1 and CLU2 gene products?" has 5 probable answers, such as \intracellular and secreted", \ER", etc. In this strategy we combined the question and the answers into a single hypothesis. For each question we created 5 hypotheses which were tested for validity using retrieval strategies adopted in our system.
In contrast to the previous runs, here we used only the background collection index, CombinedIndex, for retrieval of sentence. We limited our retrieval to only the highest scoring sentences. One of the ve hypotheses which fetched the highest scoring sentence was identi ed as the correct answer. In case of a tie we did not answer that question.
Run 7 In order to combine the retrieval results from Runs 1-5 we employed a majority-based voting to identify the correct answers. Answers that were voted most frequently as correct ones in the Runs 1-5 were selected as the correct answers for this run. In case of a tie we did not answer that question. 4.4
Other than the submitted runs, we tried various other strategies on the
training document with slightly poor performances. These runs were not included
in the submission system. For all unsubmitted runs we generated 50
pseudosentences following the hypothesis generation strategy of Run 6, combining the
query with each possible answer. The training document was (automatically)
split into individual sentences, and then both training sentences and
pseudosentences were tagged with UIMA-based Luxid R 66 Biological Entity Relation
(BER) Skill Cartridge and the Medical Entity Relation (MER) Skill Cartridge,
two rule and dictionary-based high precision taggers. We then selected the metric
that maximized the number of right answers. The steps outlined in the following
sub-sections are based roughly on the semantic search strategy used for
electronic medical record retrieval [
linguistic similarity score computed by Luxid R 6, using as features the shallow linguistic entities computed using the Luxid R 6 Analytics2 Skill Cartridge, to nd the correct answer. It returned 30% correct answers on the training set. Luxid R 6 semantic similarity In this run we used the highest semantic similarity score computed by Luxid R 6, using as features the tags provided by
the BER and MER taggers, as metrices to nd the correct answer. It returned 40% correct answers on the training set.
Luxid R 6 Crossmatch method In this run we used the highest number of
edges between the the test document and the pseudo-sentences (computed using
Luxid's Crossmatch), as a way to nd the most suitable answer. This method is
an approximation of the Literature Based Discovery method of Swanson [
trained a Nave Bayesian classi er using as features the network descriptors [
respectively. However, the best performing system, Run 6, surpasses other runs in terms of mean and median scores with a moderate standard deviation. It is also interesting to note that on average, submitted runs for reading test 2 perform considerably better than others. In this paper we proposed various strategies for automated question answering for Alzheimer disease. We selected an information retrieval based approach for this purpose. In the seven submitted runs we tried two basic approaches with some ner modi cations in each run. In the rst approach we tried a two-step retrieval process where in the rst step we select a pool of candidate sentences and in the next step we select the correct answer from this pool. This is done either by the retrieved sentence count or by the highest retrieval score. In the second approach we implemented a hypothesis generation technique using both the questions and answers of the reading tests, followed by a retrieval score based answer selection process. The most notable di erence between these two approaches is that the rst one uses only the test corpus for selecting the correct answer while the second approach uses only the background corpus. Incidentally we nd that the second approach performs considerably better than the rst one. Also a combination strategy for the various runs based on the rst approach performs worse than the second approach.
Further, we nd that there is signi cant variability in the performance of the various systems. Four di erent systems provide the top scores for the four reading tests. This shows the potential for leveraging the best results from the di erent strategies using a unifying technique, more sophisticated than the simple majority voting strategy used in one of our submitted runs.
In future work we would like to explore other semantic web-based techniques (similar to the ones outlined in Section 4.4) in aiding the performance of our retrieval-based systems. We would also like to explore other techniques for correct answer selection from the candidate answer pool. Finally, in our current implementation use of pre-processed information provided by the organizers was limited and largely out of scope. We would like to implement strategies that bene t from these information in future work.