This work is submitted to Consumer Health Information Search (CHIS) Shared Task in Forum for Information Retrieval Evaluation (FIRE) 2016. Information retrieval from any part of web should include informative content relevant to the search of web user. Hence the major task is to retrieve only relevant documents according to the users query. The given task includes further refinement of the classification process into three categories of relevance such as support, oppose and neutral. Any user reading an article from web must know whether the content of that article supports or opposes title of the article. This seems to be a big challenge to the system. Our proposed system is developed based on the combination of Keyword based features and Word embedding based features. Classification of sentences is done by machine learning based classifier, Support Vector Machine (SVM).
Natural Language Processing plays a vital role in the interpretation of human language in the most understandable format to the system. This type of role finds application in delivering the most relevant information through web search. Nowadays, information regarding health issues is one among the essential need for people. The number of researches and evidences in this field are growing rapidly day by day. So, if an individual searches through web for any health related query, a large number of documents will be retrieved. The efficiency of the search lies in the fact that the retrieved documents are relevant to the query. The main objective of proposed system is that a person irrespective of his absence of domain knowledge is supposed to get benefited through web search.
In the past years, many developments were made on
efficient retrieval of relevant clinical data. A paper was
published which discusses on the role of shared task on
overcoming barriers to NLP in clinical domain [
The proposed system is useful in finding sentences relevant to the given query and also to find whether it is a supporting, opposing or neutral sentence. An example of training data of the query ‘Skin Cancer’ is given in Table 1.
The given dataset along with some additionally collected clinical documents from Web were subjected to unsupervised feature extraction. Different approaches like Keyword based and Word embedding based information search were carried out. The integration of these two features achieved better results. Our proposed system is developed based on the combination of keyword and word embedding features. The embedding vectors with the keyword feature vector and the corresponding labels of Relevant/Irrelevant and Support/Oppose/Neutral tags were together given to train classifier. A well-known machine-learning based classifier, Support Vector Machine (SVM) is used for classification task. Section 2 includes the task description. The details about the dataset used is given in Section 3. Section 4 discusses about our proposed methodology. Experimentation and Results are explained in Section 5. The conclusion of the paper is given in Section 6. 2.
Our system is submitted in Consumer Health
Information Search in FIRE2016 [
Training data given for this task holds 5 queries Q1, Q2, Q3, Q4 and Q5 corresponding to Skin Cancer, E-cigarette, MMR-Vaccine, Vitamin-C and Women-HRT. Each query contains sentences under two categories - Relevant and Irrelevant tags (R / IR tag). For each query, the number of relevant and irrelevant sentences is different. It is further categorized into Support, Oppose and Neutral (S/O/N tag). Individual count of R/IR sentences and S/O/N sentences, for each query is tabulated in Table 2. Additional dataset related to these 5 queries were collected from online resources. Individual sentence count of additional dataset for each query is given in Table 3. Due to time-constraint, we limited the collection of additional dataset around 1000.
After analysis of given training data it has been found that, if a sentence is irrelevant to a query then most probably it will be a neutral sentence with respect to that query. At the time of generation of embedding features, training data and additional dataset is used to train the word embedding model. 4.
Our proposed system is based on the combination of Word
embedding and Keyword based generation of features. Word
embedding features are generally word vectors obtained
using the word2vec tool [
y = a + T h(D, wt−k, ..., wt+k; W ) (1) where a,T are the softmax parameters and h is the combination of word and sentence embedding features. W stands for word vectors and D stands for sentence vectors. Hence these embedding features are considered to be a feature set for the approach using word embedding model.
The second feature set in our methodology is keyword features. Keywords are extracted from the dataset given for training and testing. For the task to classify the sentences as relevant or irrelevant, keywords are extracted based on its frequency of occurrence in the relevant and irrelevant sentences. The threshold value for determining frequency of words is set as 7 for task 1 and 6 for task 2.
The list of keywords extracted for task 1 and task 2 is given in Table 4. For n keywords, a vector of length n is defined which indicates the presence or absence of the keyword as 1 or 0 respectively. The vector of length n is considered to be the keyword feature in our system. Word embedding model and keyword based model were also separately evaluated using SVM classifier.
As mentioned above, our system is developed based on the
combination of word embedding and keyword features. The
methodology of the proposed system is illustrated in
Figure 1. The sentence vectors and keyword features of each
sentence in a particular query in the training data are
combined. The Relevant/Irrelevant label from the training data
is taken. Machine learning based SVM classifier is used for
training the system [
Considering the second task, keyword features differs in this case because the keywords contributing R/IR label is different from S/O/N label. So, keywords for further classification are selected based on frequency of occurrence of keywords in support, oppose and neutral statements of training data. Therefore, to classify the sentences into Supporting, Opposing or Neutral, combined feature set which includes the embedding features, keyword features (S/O/N), labels of support, oppose, neutral of training data and predicted R/IR labels taken from task 1 are used for SVM training. The system is subjected to 10-fold cross validation while training. Training results in 5 models for 5 queries that is used for S/O/N (Support/Oppose/Neutral) classification. SVM predicts S/O/N label for test data. Table 6 tabulates the cross-validation accuracy obtained for the second task using three different approaches -Word embedding, Keyword, Keyword combined with Word embedding respectively. From the cross validation results, it is evident that the method of combination of keyword features and word embedding features is acceptable.
Results by CHIS task organizers for our proposed system is tabulated in Table 7.
The accuracy given by CHIS organizers for submission of top 6 teams for task 1 is tabulated in Table 8. Results by organizers for submission of top 7 teams for task 2 is tabulated in Table 9.
In this paper, we have proposed a methodology based on the combination of keyword and word embedding features. These features contribute in the effective retrieval of relevant information. Keyword features for any set of document can be extracted based on its frequency of occurrence. The proposed system will be helpful in extracting the most relevant document for a query, among a large pool of documents in web. Irrespective of the position we have acquired in task 1, our accuracy value is comparable to that of others. The second task is more challenging due to further classification. By considering sentimental features for sentences, accuracy can be increased.
We would like to express our sincere gratitude to the organizers of Forum for Information Retrieval Evaluation 2016 for organizing a task with great scope of for research. We would also like to thank Xerox Research Centre, for organizing the CHIS task.