In this paper, we describe our participation in the 2015 BioASQ challenge on Bio-Medical Question Answering. For Question Answering task (Task 3b), teams were provided with natural language questions and asked to retrieve responses from PubMed corpus in the form of documents, snippets, concepts and RDF triplets (Phase A) and direct answers (Phase B). For Phase A, we took the support of PubMed search engine and our snippet extraction technique. In our QA system, apart from the standard techniques discussed in literature, we tried the following novel techniques to - a) leverage web search results for improving question processing and b) identify domain words and de ne a new answer ranking function based on number of common domain words. We scored an F-measure of 0.193 for document extraction and F-measure of 0.0717 in snippet generation.
The present innovations in the bio-medical domain is leading to the creation
of large amounts of data. The bio-medical literature growth can be understood
from the vast data in PubMed [
QA has been a well studied research area [
In this paper, we describe the approach taken by the IIIT-Hyderabad team for Task 3b of BioASQ Challenge. For retrieving documents as answers, we used chunking, stop word removal and search query formulation techniques. Later, from the top documents, we lter the most relevant phrases as snippets. For extracting the exact and Ideal answers from the snippets, we used cosine similarity and noun chunk identi cation techniques.
The rest of the paper is organized as follows: Section 2, describes the previous work done in Bio-Medical QA. Section 3 describes our approach in more detail. Section 4 gives the experimental results. Finally, Section 5 concludes the paper with future work. 2
The Bio-medical Question Answering has been a challenge from past few years. There has been not much progress since then. The major challenge for this was a very complex domain. So, only domain expert could understand the inner details for system to be built. Major concentration was done on the factoid based questions and yes/no questions.
MedQA [
The jikitou [
OHSU (Oregon Health & Science University) [
In [
Our current approach di ers with the above approaches in the following ways: a) We leverage web search results in question processing and b) We de ne new similarity metrics based on common domain words. 3
We have designed an algorithm to extract the high informative documents for a given question from PubMed articles. 3.1
The architecture of the system is shown in Figure 1 3.2
This Algorithm takes a bio medical question 'Q' and outputs a set of 10 PubMed documents which are having high probability to contain answer 'A' for given Q. Detailed explanation of each step of algorithm is given below. 1. Question Processing: Let the given question be Q, we need to process the question to make it e cient and optimized for searching. For this we have sequence of sub steps which are explained below. (a) Cleaning: We clean the question for making the search e cient. In this step all unnecessary symbols like question mark(?) , dot(.) etc are removed. We have found that (-) makes the chunking task little di cult and wrong which is crucial task in this step. So Hyphens are replaced with some Named Entity Words.
Algorithm 1 Domain Word Extraction Algorithm 1: N N um Of T op Results(20) 2: domainU rlP atterns f\nlm:nih:gov"; \webmd:com"; \medicine"; \biocare"; \drugs"; :::g 3: function DomainWordIdentification(chunk; domainU rlP atterns) 4: topResults SearchAPI(chunk; N ) 5: f ocusW ord null 6: for all result 2 topResults do 7: if result 2 domainU rlP atterns then 8: f ocusW ord chunk 9: else 10: remove(chunk) 11: end if 12: end for 13: return f ocusW ord 14: end function 15: function SearchAPI(chunk; N ) 16: results GoogleAP I=BingAP I(chunk; N ) 17: return results 18: end function (b) Chunking: We need to do chunking to get the phrases(chunks) to the modi ed Question Q. These chunks will be very useful to avoid removal of important(Focus) words which will be done in the next step. We used Annotator module from NLTK3 Package for chunker. (c) Stop Word Removal: The chunks with all the stop words will be removed. As they don't help at all in a keyword based search. This step makes the question more optimized. We have used NLTK corpus English StopWords list for this task. (d) Domain Word Identi cation: In the processed query sentence Q, there will be generic words which may not be stop words. But, they contribute nothing in getting the relevant documents. Focus Word Identi cation step nds only the chunks which are Domain-words, Important Words. The pseudocode for identifying the focus word/chunk is shown at Algorithm 1.
We have observed the following list of url patterns that are most relevant to Bio-Medical domain4.
After the question processing step, the question Q will be modi ed into set of Focus words, namely Set FQ. 2. PubMed Search: The words in the Focus Word Set will be combined(concatenated) to make a single string. This string will be red/searched in PubMed search engine and top 200 documents will be retrieved. 3 Natural Language Toolkit http://www.nltk.org/ 4 Patterns in website urls which store bio-medical information \nlm.nih.gov", \webmd.com", \medicine", \biocare", \drugs" Algorithm 2 Document Re-Ranking Algorithm 1: Q Query 2: relDocs
relaventDocuments 3: function RankAllDocuments(Q; relDocs) 4: Q RemoveStopWords(Q) 5: scores fg 6: for all doci 2 relDocs do 7: Ti doci:title 8: Ti RemoveStopWords(Ti) 9: scores[i] CosineSim(Q; Ti) 10: end for 11: scores; topDocs 12: return topDocs 13: end function
SortScores(scores; relDocs)
3. Document Re-Ranking: As our question processing is not a standard
one, we don't completely depend on PubMed ranking of documents. So we
rank the obtained 200 documents again with our approaches.
(a) Cosine similarity [
We take the original query Q, and Document Title T. (b) Existence Test Score: This measure tells number of common Focus words (Domain words) present in Question Q and title of the document Ti.
Scores[i] = ExistenceT estScore(Q; Ti) (1) (c) Hybrid Approach: Finally, we combined both the approaches giving them separate weights and interestingly the scores were better. In this approach the score[i] will be de ned as below,
Score[i] =
(2) ; are normalization constants.
After this step, high ranked which are most relevant 10 documents to the given query Q will be retrieved and output to the system. 3.3
This section explains about retrieving top 10 snippets for given question Q. The algorithm for Snippet Generation and Ranking is shown in Figure 2
Initially we take the query Q and send in to the`Document Retrieval algorithm. From this step, we will get top 10 documents for the query Q.
After obtaining top 10 most relevant documents (D1...D10 ) we take abstracts of all those documents. From all the abstracts we extract all the sentences and make a set S.
For all the sentences s in S, we compute similarity scores with query Q. After nding similarity scores for all the sentences we sort and take top 10 sentences matching most with the query Q. We call those High-Matching sentences and snippets and output them to the system.
As we have parsed the PubMed web(search) page to get the abstracts and pmids, we have used Regular Expressions to identify abstracts and also to split sentences correctly. While nding similarity score between sentences s and query Q, we have used above explained 3 approaches and found hybrid approach(Cosine similarity score+Existence Test score) performance was good.
This algorithms gave good snippets in which always at least 4 of them contained the exact answer.
Here a Question and related snippets will be given to our algorithm and it should nd Ideal answer i.e. a one or two line answer which perfectly answers the given query.
For this task we have taken all the sentences from the given snippets and calculated similarity scores with the approaches explained in section 3.2. Top 10 Ideal answers with highest similarity scores will be returned. In this section we will be given with a query and set of snippets. We should result the exact one or two word answers. We have designed an algorithm for giving exact answers for Factoid questions. That approach is explained as a ow chart in Figure 3.
As shown in the Figure 3, we start with query Q and all snippets S. We apply, Phase A algorithm for nding top Ideal answers. For each, Ideal answer we nd chunks and we consider only \Noun chunks" as we are dealing with only Factoid questions. Each time, we modify the given query Q by replacing question words (like what, when, who etc. ) with the considered \Noun chunk", then we name that new query as Q. Then, we nd the similarity score between Q and Ideal answer Idi. Similarly, we repeat the same procedure for all ideal answers and their noun chunks. At the end, we sort the similarity scores and output top 10 related noun chunks as exact answers. In this algorithm, we used NLTK parser and Cosine similarity measurement. We got good results with proposed approaches.
As part of the BioASQ 3b challenge 2015, we have participated in the batch wise submissions. We could perform relatively better than the baseline System. These results of all submissions for documents and snippets are shown in Table 1 and Table 2. 4.1
Task 3b of BioASQ 2015 released a training dataset [
The evaluation metrics used in this task are mean precision, mean recall, and
mean F -measure of the documents, snippets, exact and ideal answers returned
by the system [
Di erent experiments have been conducted to improve the accuracy of system such as: 1. Increased the retrieved documents after PubMed Search step from 60 documents to 100 documents. 2. De ned new similarity measure called Existance Similarity and combined it with cosine similarity, which increased the accuracy of the system.
Our best results in snippet extraction for Task 3b Phase A has been achieved(see table 2, bold). Our system name for submission was \qaiiit system 1". In fact that was achieved in statistical approach. This approach was found better compared to baseline system.
In Table 3, from the snippets given by the BioASQ, we found most relevant snippet based on hybrid similarity match and returned respective snippet as the ideal answer. By using this method, rougue score was about 0.303, just below the baseline of 0.46. 5
In this paper, we describe the approach taken by IIIT-H team for the Bio-Medical Question Answering task of BioASQ task 3B. Apart from the standard QA techniques, our current approach di ers in the following ways - a) We leverage web search results in question processing and b) We de ne new similarity metrics based on common domain words. By applying our approach, we obtained Fmeasure of 0.193 for document extraction and F-measure of 0.0717 in snippet generation.
As part of the future work, we would be working on the Triples extraction, which is still under progress, as no one has even attempted it. We will also be using more domain speci c entity recognition tools, for more cleaner way of identifying the exact answer. For the ideal answer or summary type answers, we are working on answer generation techniques from snippets of multiple documents.