People are increasingly turning to the World Wide Web to nd answers for their health and lifestyle queries, While search engines are e ective in answering direct factual questions such as `What are the symptoms of a disease X?', they are not so e ective in addressing complex consumer health queries, which do not have a single de nitive answer, such as `Is treatment X e ective for disease Y?'. Instead, the users are presented with a vast number of search results with often contradictory perspectives and no de nitive conclusion. The term \Consumer Health Information Search" (CHIS) is used to denote such information retrieval search tasks, for which there is \No Single Best Correct Answer". The proposed CHIS track aims to investigate complex health information search in scenarios where users search for health information with more than just a single correct answer, and look for multiple perspectives from diverse sources both from medical research and from real world patient narratives.
World Wide Web is increasingly being used by consumers
as an aid for health decision making and for self-management
of chronic illnesses as evidenced by the fact that one in
every 20 searches on Google [
It is pretty straightforward to get an answer to the query \what are the symptoms of Diabetes" from the search engines. However retrieval of relevant multiple perspectives for complex health search queries which do not have a single de nitive answer still remains elusive with most of the general purpose search engines. For example, a user health query such as \can metabolic therapy cure brain cancer" causes considerable frustration for the searcher as he needs to wade through hundreds of search results to obtain a balanced view of the diverse perspectives/points of view available, both for and against the hypothesis posed in the search query. Subjective health related queries such as 'does treatment X e ective for disease Y?' or 'can X cause disease Y' do not have a single de nitive answer on the web due to the multiple supporting/opposing perspectives available on the web related to them, instead multiple perspectives (which very often are contradictory in nature) are available on the web regarding the queried information. The presence of multiple perspectives with di erent grades of supporting evidence (which is dynamically changing over time due to the arrival of new research and practice evidence) makes it all the more challenging for a lay searcher. Figure 1 depicts the precise scenario. In our Consumer Health Information search CHIS shared task track on FIRE 1, we have attempted to encourage the development of innovative computational models to statistically represent the multipleperspective around a general health search query and therefore, assist the self-searcher with better and meaningful information insights.. 2.
At present times, there has been considerable interest
in the eld of stance classi cation and stance modelling.
Stance classi cation has been applied to di erent debate
settings such as congressional debates [
Unlike many of the earlier research settings which have
analyzed posts on public debate topics, multi-perspective
con1https://sites.google.com/site/multiperspectivehealthqa/
sumer health information is not typically characterized by
strong emotion/opinion bearing language, nor does it have
strongly delineated supporting/opposing topic words. They
typically contains domain speci c technical terms and sparse
in emotional/a ective words and is typically factual in
nature. A closely related work [
S1: Many studies have found that skin cancer rates are increasing in indoor workers.
S2: very few studies have demonstrated that skin cancer rates are increasing in indoor workers.
Both sentences contain the topical phrase skin cancer rates in indoor workers with sentence S1 providing evidence in support of it, whereas sentence S2 providing evidence opposing it. This illustrates the di cult of identifying stance delineating topic words in health related text.
The technical language of the information in these queries is also another factor which makes the stance classi cation complex. Given an example sentence E-cigarettes contain di-acetyl which has been associated with popcorn lung syndrome for a sample query E-cigarettes are safer than normal cigarattes, it is not evident at rst glance, whether this sentence is supportive/opposing the query. This makes the task more challenging, compared to general domain stance classi cation.
Given a CHIS query, and a document/set of documents associated with that query, the task is to classify the sentences in the document as relevant to the query or not. The relevant sentences are those from that document, which are useful in providing the answer to the query. These relevant sentences need to be further classi ed as supporting the claim made in the query, or opposing the claim made in the query.
Example query: Does daily aspirin therapy prevent heart attack?
S1: \Many medical experts recommend daily aspirin therapy for preventing heart attacks in people of age fty and above." [a rmative/Support]
S2: \While aspirin has some role in preventing blood clots, daily aspirin therapy is not for everyone as a primary heart attack prevention method." [disagreement/Oppose] 3.1
There are two sets of tasks: 1. TASK A: Given a CHIS query, and a document/set of documents associated with that query, the task is to classify the sentences in the document as relevant to the query or not. The relevant sentences are those from that document, which are useful in providing the answer to the query. 2. TASK B: These relevant sentences need to be further classi ed as supporting the claim made in the query, or opposing the claim made in the query.
Example : Query- \Are e-cigarettes safer than normal cigarettes?"
Retrieved sentence S1 - \Because some research has suggested that the levels of most toxicants in vapor are lower than the levels in smoke, e-cigarettes have been deemed to be safer than regular cigarettes". A)Relevant, B) Support
Retrieved sentence S2 - \David Peyton, a chemistry professor at Portland State University who helped conduct the research, says that the type of formaldehyde generated by e-cigarettes could increase the likelihood it would get deposited in the lung, leading to lung cancer." A)Relevant, B) Oppose
Retrieved sentence S2 - \Harvey Simon, MD, Harvard Health Editor, expressed concern that the nicotine amounts in e-cigarettes can vary signi cantly." A)Irrelevant, B) Neutral
Our task have 5 consumer health queries, Figure 2 and gure 3 below presents the comprehensive statistics of the CHIS queries used in our task released as training and test respectively.
A total of 9 teams participated in task and 9 submissions are obtained against Task A and 8 submissions are obtained against Task B. Details of the participating teams are shown in gure 4 below.
As can be observed in Task A, team SSN NLP and team Fermi have secured the top scoring positions with accuracy 78.10% and 77.04% respectively. SSN NLP has proposed a decision tree model based on sophisticated text features including part-of speech. They have used a chi-square feature selection to extract the informative features and reduce the number of spurious features and demonstrated that such a feature selection approach can o er a signi cant gain. Team Fermi have used a deep neural network architecture with Recti ed-linear (ReLu) and Sigmoid activation over bag-ofphrase features.
Team JU KS group and Techie-challengers have secured
the second position jointly with a closed call of 73.39% and
73.03% accuracy respectively. JU KS group has implemented
a support vector machine with polynomial kernel to classify
the data. They have curated informative text features such
as part-of-speech matching, neighborhood matching to
represent the input data. Techie-challengers has proposed a
naive-bayes classi er on doc2vec [
With accuracy 70.20% and 70.28%, team Amrita Cen and
individual participant Jainisha Shankhavara have ranked third
jointly. Team Amrita Cen has used a support-vector-machine
classi er on top of input feature representation obtained by
word-embedding and keyword generation techniques.
Jainisha has proposed classi cation model based on BM-25 [
Hua Yang has approached the task from the perspective of improving understandability in consumer health related searches and their information retrieval based query expansion module has provided a 69.33% accuracy.
Team Amrita Fire Cen has used a random forest classi er on distributional semantic representation of the input and obtained 68.12% accuracy. They have used the non-negative matrix factorization technique for obtaining the distributed 2This is the nal updated result table, the individual team working notes may not contain the latest updated version due to some late changes representation.
Team JNUTH model uses a aggregate over a range of similarity measures to obtain the relevance-irrelevance decision for a data input. They have obtained 54.84% accuracy. 4.2
In task B, team JNUTH has jointly secured the rst position with team Fermi. JNUTH has used a C-support vector machine classi er with radial basis kernel. They have used tf-idf for input representation followed by a max-feature sorting. Their model has obtained 55.43% accuracy. Team Fermi has used a deep neural network architecture and a bag-of-phrase representation to achieve 54.87% accuracy.
With a score of 53.99% Hua Yang have secured the second rank. His model uses a naive-Bayes classi er and tf-idf representation. Team Techie-challengers also used a naiveBayes classi er, but on doc2vec input representation to obtain 52.47% accuracy. Therefore, they hold the third rank.
Team Amrita Fire Cen has used a random forest classi er on distributional semantic representation of the input and obtained 38.53% accuracy. Individual participant Jainisha Sankhavara has developed a model based on BM-2 ranking function to obtain overall accuracy 37.96%.
Team Amrita Cen has modeled using a support vector machine classi er with input feature representation obtained by word-embedding and keyword generation techniques to obtain 34.64% accuracy. Team JU KS group has modeled the task as sentiment classi cation problem and their innovative feature set consists of positive, negative and neutral polarity words along with information from Task A. They have achieved an overall accuracy of 33.64%. 5.
We thank all the participants for expressing interest in our track. It has been a great experience to witness the innovative models and techniques proposed by di erent teams. The CHIS task was surely a challenging one with little presiding literature and yet, as can be observed from the previous section, in both the tasks there are closed calls in terms of the performances of di erent teams.
We also express our sincere gratitude to the organizing and program committee of Forum for Information Retrieval Evaluation (FIRE), 2016, especially Mr. Parth Mehta, for providing us with the opportunity to hold the shared task and to connect with the enthusiast researchers across India and abroad who share the same interest.
In future, we are looking forward to work again with such expert groups to come up with novel solutions to more challenging health-care data analytic problems. 6.