In this work we present a novel approach towards the extraction of factoid answers to biomedical questions. The approach is based on the combination of structured (ontological) and unstructured (textual) knowledge sources, which enables the system to extract factoid answer candidates out of a predefined set of documents that are related to the input questions. The candidates are scored by applying a variety of scoring schemes and are combined to find the best extracted candidate answer. The suggested approach was submitted in the framework of the BioASQ challenge1 as the baseline system to address the automated answering of factoid questions, in the framework of challenge 1b. Preliminary evaluation in the factoid questions of the dry-run set of the competition shows promising results, with a reported average accuracy of 54:66%.
The task of automatically answering natural language questions (QA) dates back to the
1960s, but has only become a major research field within the information retrieval and
extraction (IR/IE) community in the past fifteen years with the introduction of the QA
Track in TREC2 evaluations in 1999 [
The BioASQ challenge [
the semantic information that can be extracted from the aforementioned resources e.g., MetaMap.
Under this scope, in this paper we present a QA system that can address with high
accuracy the factoid questions in the biomedical domain. The system performs two
sequential processes; Question Analysis, and Answer Extraction which are explained in
detail in Section 2. It uses all of the MEDLINE indexed documents as a textual
corpus, the UMLS metathesaurus as the ontological resource, and ClearNLP4, OpenNLP5,
and, MetaMap as additional tools to process the corpus. The novelty of the work lies
in the combined application of several different scoring schemes to assess the extracted
candidate answers. In the same direction with the current work, the use of (weighted)
prominence scoring and IDF scoring has been widely applied in the past [
Initially, a question analysis module analyses the question, by identifying the lexical answer type (LAT), given a factoid or list question. The LAT is the type of answer(s) the system should search for. Next, documents relevant to the question are retrieved by the document or passage retrieval module, which takes the input question and transforms it into a keyword query. For this step, we rely on the results provided by the GoPubMed search engine6. In a last step, the top N retrieved documents are being processed by the answer extraction module. This module finds possible answer candidates within the relevant documents and scores them. These scores are combined to a final score which serves as the basis for ranking all answer candidates. 2.1
Question Analysis is responsible for extracting the LAT (lexical answer type), which is crucial for understanding what the question is actually asking about. In the English language questions follow specific patterns, thus making it easy to apply pattern based
extraction methods. However, there are different kinds of factoid questions, like the following examples illustrate: Example 1. What is the methyl donor of DNA (cytosine-5)-methyltransferases? Example 2. Where in the cell do we find the protein Cep135? Example 3. Is rheumatoid arthritis more common in men or women?
From this perspective, the questions answerable by the current system fall into one of the three following classes:(1) what/which-questions (Example 1), (2) wherequestions (Example 2), and, (3) decision-questions, (Example 3). Factoid or list questions usually carry information about the type of the answer (the LAT). The phrase containing the LAT is found directly after the question word (what/which), or after the question word followed by a form of “be”. Where-questions do not have an explicitly defined LAT; instead, they implicitly expect spatial concepts as answers. There are also cases in which where-questions restrict the set of potential answer types further, by defining a place where the (spatial) answer concept should be part of, e.g. “the cell” in Example 2. Decision-questions also do not have a LAT. It rather consists of a number of possible answers, from which one has to be selected, e.g., “men”, “women”. Furthermore they define a criterion by which the answer candidates have to be differentiated, e.g., “more common”. The extraction rules for all types of questions are based on the chunks and the dependency parse of the question. For instance Example 1 falls into the following pattern: ‘NP[WhatjWhich] VP[BE] NP[*] *?’, where * serves as a wildcard, NP stands for nounal phrase and VP for verbal phrase. The LAT of questions falling into this pattern can be found in the second NP. 2.2
After the relevant documents for the question have been retrieved, all annotated concepts of these documents are considered possible answer candidates. To extract the right answer the candidates are scored and ranked. The proposed system supports 6 different scores of 3 different types (prominence, type, specificity/idf ). In the following, the different scores are explained in detail. The following notation is utilized: q is the input question, ac is a candidate answer, D is the document set that is relevant to the question, S is the set of all sentences in D, and A is the set of all documents present in the used corpus (MEDLINE).
Prominence Scoring Prominence Scoring is based on the hypothesis that the answer to a question appears often in the relevant documents. Thus, this score counts all occurrences of a concept within the sentences of all of the relevant documents, and can be formalized as follows: scorepr(ac) =
Ps2S I(ac 2 s) jSj ; (1) where I is the indicator function, returning 1 if the argument is true and 0 otherwise. The problem with this score is the assumption that each sentence is equally relevant to the question. This is a very poor assumption giving rise to the refined hypothesis that the answer to a question appears often in the relevant sentences. The refined hypothesis can be formulated by weighting sentences according to their similarity to the question, as shown in the following equation: scorewpr(ac) =
Ps2S sim(q; s) I(ac 2 s)
Ps2S sim(q; s)
The weighted prominence score (wpr) makes use of a similarity measure for sentences.
The way it is implemented in the proposed system is by measuring the number of
common concepts between the question and a sentence, normalized by the number of
question concepts. This measure has also been used in other QA systems [
A
j j
a2A I(ac 2 a)
)=log(jAj)
The textual type scorer tries to find 3 types of connections between the LAT and the
answer candidate. The most obvious pattern is the one where the answer candidate is a
subject and the the LAT is an object, connected via a form of “be”. Two other syntactic
Type Coercion Type Coercion is based on the hypothesis that the type of the answer
aligns with the lexical answer type (LAT). Type coercion was introduced by IBM within
the Watson system [
The UMLS type score is based on the UMLS semantic network. Each UMLS concept of the metathesaurus has a set of semantic types assigned to it. These semantic types are hierarchically structured in the semantic network. If the question analysis module finds the LAT to be a UMLS concept, the semantic types of this concept and its children in the hierarchy of the semantic network become the target types of the question. If the candidate answer’s semantic types and the target types have common types, the UMLS type score will be 1, and 0 otherwise. This is shown analytically in the following equation: scoreumls = (1 if sem types(ac) [ sem types(LAT ) 6= 0 else (2) (3) (4) constructions, namely nominal modifiers and appositions, are used to infer the type of the answer candidate. The following examples illustrate the 3 cases: – . . . naloxone is standard medication. . . [BE] – . . . the medication naloxone. . . [NOMINAL MOD.] – . . . naloxone, a medication. . . [APPOS] All of the above examples are evidence for naloxone being a medication. However, nominal modifiers and appositions are not that reliable, so the textual type score for these cases should be lower. The scoring is described analytically in the following: scorepa = If there is no textual evidence in the retrieved documents D, there could still be this kind of evidence in some other MEDLINE abstracts (A). This kind of evidence is called supporting evidence, because it searches also in unrelated documents for evidence. The following equation shows how the supporting evidence is scored: Eventually, in the proposed system the top 5 retrieved supporting documents for each answer candidate are examined.
Score Merging After scoring the answer candidates, the scores have to be combined into a final score. In the proposed system, we followed a supervised approach to learn appropriate weights for the individual scores. Given a dataset of factoid questions and their “gold” answers, a training set consisting of positive (right answers) and negative (wrong answers) examples represented by their scores is constructed in order to train a logistic regression classifier. Hence, the training instances are answers, and the features are the scores of the individual measures. The logistic regression learns appropriate weights for each of the represented features (scores). The learned weights of the output classifier (learned model) are then used as the weights for each of the scoring measures respectively. The final score is produced by applying the learned logistic regression formula.
Candidate Selection and Answer Suggestion For factoid questions the candidate with
the highest final score is chosen to be the answer. However, in the case of list questions it
is not easy to define the right cut-off in the list manually, e.g., how many items to suggest
as answer starting from the beginning of the list. For this purpose, we employed again a
supervised approach, where the threshold on the final score is estimated by calculating
the F1 score on all training list-questions for all possible thresholds within a certain
interval. The threshold with the highest F1 score is finally chosen to be the cut-off for
list questions.
Experiments were conducted on the dry run test set of the BioASQ Challenge 1b [
The results of the experiments, though preliminary in the sense that they were conducted in the dry-run set of the BioASQ challenge, indicate that the suggested baseline approach is able to achieve reasonable performance on answerable factoid questions. However, there is still a lot of room for improvements, like considering more scores and making the system faster to handle all supporting evidence. More sophisticated systems for factoid question answering, similar to the IBM Watson system, but especially designed for the biomedical domain, can be developed given the huge amount of structured and unstructured resources in the domain, but this requires large infrastructure, since QA pertains to nearly every aspect of information extraction and natural language processing. In addition, the effect of adding larger number of training questions will be investigated in the future, once the benchmark questions from the first BioASQ challenge are released to the public. The proposed system was submitted as a baseline for challenge 1b of the BioASQ competition to address exclusively factoid questions, and as a future work, we plan to analyze the results on the actual test set of the competition and study how further improvements may be achieved in an effort to address more effectively factoid questions in the biomedical domain.