We brie y outline the YodaQA open domain question answering system and its initial adaptation to the Biomedical domain for the purposes of the BIOASQ challenge (question answering task 3b) on CLEF2015. The YodaQA system for open domain factoid English question answering has been published recently. [1] [2] The system is a fully open source, modular pipeline inspired by the IBM Watson DeepQA system [3]. So far, the system has not been specialized for any particular domain, this represents the rst such e ort. The BIOASQ challenge [4] aims at semantic indexing and question answering in the biomedical domain using a variety of knowledge bases from the given domain. The BIOASQ 2015 has tasks 3A and 3B, where 3A concerns semantic indexing and 3B is about question answering, where we participated. The BIOASQ Task 3B is further split into phase A (information retrieval) and phase B (answer production); the phases are evaluated separately, with gold standard phase A results (documents, snippets and triples) available for phase B. Our system participated in the phase B evaluation. The paper is structured as follows. In Sec. 2, we brie y outline the YodaQA system in its original form. In Sec. 3, we discuss the changes of the system for the biomedical domain. In Sec. 4, we review the system performance.
{ Question Analysis extracts natural language features from the input and produces in-system representations of the question. We currently build just a naive representation of the question as a bag-of-features. The most important characterization of the question is a set of clues (keywords, keyphrases and concept clues that exactly match enwiki titles) and possible lexical answer types. { Answer Production generates a set of candidate answers based on the question, typically by performing a Primary Search in the knowledge bases according to the question clues and either directly using search results as candidate answers or ltering relevant passages from the result text (the Passage Extraction) and generating candidate answers from the ltered passages (the Passage Analysis). Answers are produced from text passages by a simple strategy of considering all named entities and noun phrases. { Answer Analysis generates various answer features based on detailed analysis. Most importantly, this concerns lexical type determination and coercion to question type. Other features include distance from clues in passages or text overlap with clues. { Answer Merging and Scoring consolidates the set of answers, removing duplicates and using a machine learned classi er (logistic regression) to score answers by their features. 3
We made a variety of adjustments to t our end-to-end pipeline to the BIOASQ task. The changes are available within the public open source code base (https: //github.com/brmson/yodaqa) in the d/clef2015-bioasq branch.
As a minor technical change, we enhanced our question analysis for imperative and otherwise speci cally phrased questions which were uncommon in our TREC-based open domain dataset.
Our system is designed to answer just factoid questions, while the BIOASQ challenge also includes list and yes-no questions. For the list question, we simply use the top 5 answer candidates returned by the system. Since we implemented no text entailment algorithm yet and there was no easy way to skip yes-no questions, we simply use a xed yes answer for all since it was signi cantly more prevalent in the training dataset.
Similarly, our system is designed to return narrow answers, not sentencelength answers that include background and justi cation. Therefore, we always return empty string as the ideal answer and supply our answers as the exact answer. However, the BIOASQ de nition of exact answers might be more stringent than ours, which simply requires that the gold standard answer is a sub-string of the produced answers answer (e.g. \the red color" would be acceptable for gold standard \red" in our scenario). Therefore, we modi ed our system to require exact matches during training, and disabled a heuristic in answer analysis which attempts to nd a focus word in the answer and run analysis (like title lookup, type coercion) on it instead of the whole answer.
As we focus on the phase B of the question answering task, we do not perform an explicit primary search on questions and instead base answer production on the search result snippets (generated by phase A) supplied along with the question on program input. These snippets are equivalent to passages our primary search would produce.1
To further improve the accuracy of the system, we implemented an
enhanced answer production strategy (inspired by the Jacana QA system) that
approaches the problem of identifying the answer in a text passage in a way
similar to named entity recognition: as a (token) sequence tagging (by
begin-insideoutside labels) that uses the conditional random eld model to predict labels. [
A crucial feature for scoring answers is information on successful type
coercion. This involves identi cation of Lexical Answer Type (LAT) in the question
(typically an easy task using a few xed heuristics), production of a set of LATs
describing the answer and the question-answer type coercion using straight string
matching and Wordnet [
To evaluate the performance of our system, we split the (randomly reshu ed)
reference training dataset provided for the Task 3B to a local dev/train set
(100 questions) and a test set (100 questions).3 For comparison, we also include
baseline version performance4 on the \curated" factoid open domain dataset [
nal w/o GeneOntology w/o G.O., CRF
open domain nal, ignoring yes/no q. 43.5% 33.0% 33.0% 33.0% 79.3%