In this paper we describe a method to determine the relevancy of a query with a sentence in the document in the field of medical domain. We also describe a method to determine if the given statement supports the query, opposes the query or is neutral with respect to the query. This is a part of CHIS shared task at FIRE 2016.
World Wide Web is increasingly being used by consumers as an aid for health decision making and for self-management of chronic illnesses as evidenced by the fact that one in every 20 searches on google is about health. Information access mechanisms for factual health information retrieval have matured considerably, with search engines providing Fact checked Health Knowledge Graph search results to factual health queries. It is pretty straightforward to get an answer to the query “what are the symptoms of Diabetes” from the search engines. However retrieval of relevant multiple perspectives for complex health search queries which do not have a single definitive answer still remains elusive with most of the general purpose search engines. The presence of multiple perspectives with different grades of supporting evidence (which is dynamically changing over time due to the arrival of new research and practice evidence) makes it all the more challenging for a lay searcher.
We use the term “Consumer Health Information Search” (CHIS) to denote such information retrieval search tasks, for which there is “No Single Best Correct Answer”; Instead multiple and diverse perspectives/points of view (which very often are contradictory in nature) are available on the web regarding the queried information. The goal of CHIS track is to research and develop techniques to support users in complex multi-perspective health information queries.
Given a CHIS query, and a document/set of documents
associated with that query, the FIRST task is to classify the
sentences in the document as relevant to the query or not [
For the shared tasks described above, we adopt a deep learning
approach for solving them. Deep learning is a method which
allows computers to learn from experience and understand the
world in terms of a hierarchy of concepts, with each concept
defined in terms of its relation to simpler concepts. By gathering
knowledge from experience, this approach avoids the need for
human operators to formally specify all of the knowledge that the
computer needs. The hierarchy of concepts allows the computer to
learn complicated concepts by building them out of simpler ones.
We use a deep neural network to train the sentences.
The problems described above are modeled as a supervised
learning task [
We have selected binary bag-of-phrases [
We use a deep neural network for training for both the tasks. The
input layer had as many neurons as the input features. Task 1 is a
binary classification problem, indicating if the sentence was
relevant to the query or not. Task 2 is a multi-class classification
problem, which indicates if the sentence supports, opposes or is
neutral to the query. Table 3 shows the architecture of the neural
network for both of the CHIS tasks [
For task 1, the classification is a binary classification problem with a binary cross entropy layer at the output. For task 2, it is a multi-class classification problem, and hence a softmax layer is used at the output layer. For training the deep neural network, we used keras. Keras is an open source neural network library written in Python. It is capable of running on top of either Tensorflow or Theano. Designed to enable fast experimentation with deep neural networks, it focuses on being minimal, modular and extensible. We train both the neural networks for 150 epochs for convergence.
The following are the results obtained on the test set. Table 4 shows the average precision, recall and F1 score of the classifier for task 1. Table 5 shows the average precision, recall and F1 score of the classifier for task 2.
Predicting the relevance and determining if a sentence supports the given query is not a trivial problem and needs knowledge of Natural Language Processing and Information Retrieval techniques. In this paper we proposed a fast deep learning method to predict the same using a deep neural network. We observe that the average precision for task 1 is 77.03% and for task 2 is 54.86%. Task 2 is a multi-class problem and is more difficult than task1.
In this paper, we have used a select set of phrases as features.
Since the sentences and the query, both are short text segments,
features using Natural Langauge Processing like POS tagging etc
can be used as features augmented with the existing features to
improve the precision and recall [
All the code is available at https://github.com/saradhix/chis for research and academic purpose.