The paper reports our experiments in tackling the CLEF 2012 Pilot Task on Machine Reading for Question Answering. We introduce the technique of index expansion, which relies on building a search index enriched with information gathered from a linguistic analysis of texts. The index provides a highly tangled representation of the sentences where each word is directly connected to others representing both meaning and relations. Instead of keeping the knowledge base separate, the relevant knowledge gets embedded within the text. We can hence use efficient indexing techniques to represent such knowledge and query it very effectively. We explain how index expansion was used in the task and describe the experiments that we performed. The results achieved are quite positive and a final error analysis shows how the technique can be further improved.
The paper reports our experiments in tackling the CLEF 2012 Pilot Task on Machine Reading for Question Answering. The task aimed at exploring the ability of a machine reading system to answer questions about a scientific topic, in particular about the Alzheimer disease, using knowledge extracted from biomedical texts.
The evaluation has the format of traditional Multiple Choice Reading Comprehension tests: it involves reading a single scientific article at a time and answering a set of questions regarding information that is stated or implied in the text. Multiple choice answers are provided for each question, each having five options with only one being correct. A background collection of reference articles about the topic is provided by the organizers, which may be exploited for learning knowledge useful in answering the questions. Nonetheless the principal answer is to be found among the facts presented in the given test document.
The rationale of the task [
We approach the task using a technique that we call “index expansion”, which is the dual to query expansion. Query expansion adds terms to the query in order to achieve better recall, but this often results in poorer precision, since, because of term polysemy, the terms introduced may have also quite different meaning and may match irrelevant documents, introducing a lot of noise in the results.
Index expansion instead keeps the original query terms but adds variants of terms to the index, for example synonyms and hyperrnyms. This introduces much less noise, since if the variant term is used in the query, then it is relevant and the match will be successful. On the other hand, inappropriate terms will not be used in the queries and hence their presence in the index will not affect negatively the results. The only drawback of the approach is that the size of the index increases, but this is a minor issue, since the index is compressed and disk space is cheap. In other words we trade space for speed and accuracy, which is akin to the technique of “database denormalization”.
The approach extends the one that we proposed and applied successfully to the task
of Blog Opinion Mining at TREC 2006 [
The index is enriched with information extracted from linguistic analysis of the documents. The index in fact can be seen as a multilayer index, where each layer represents one kind of annotation. The layers are: ─ form ─ lemma ─ POS ─ head ─ deprel ─ lemma ─ synonyms ─ hypernyms the original text the lemma of each word the POS of each word the governor of each word the dependency relation of each word with its governor the lemmas of each word list of synonyms for each word list of hypernyms for each word The process of answering a question relies on a similar analysis of text: questions are parsed and from the parse tree a basic query in the DeepSearch query language is generated. To each of these queries one of the multiple possible answers is added. Those queries which obtain an answer with the highest score are considered to include the correct answer. 2
The collection of background documents about Alzheimer's Disease provided for the task included: around 66,000 abstracts from PubMed1; around 8,000 Open Access full articles from Central2; full articles about the key hypotheses in Alzheimer Disease published by Elsevier.
The documents were provided also in a preprocessed format, split into sentences
and tokens, each one annotated with lemma, POS, dependency parsing annotations
1 http://www.ncbi.nlm.nih.gov/pubmed/
2 http://www.ncbi.nlm.nih.gov/pmc/
from the dependency parser GDep [
The idea of index expansion is to create a highly tangled representation of the sentences where each word is directly connected to others representing both meaning and relations. Instead of keeping the knowledge base separate, the relevant knowledge gets embedded within the text. We can hence use efficient indexing techniques to represent such knowledge and query it very effectively with suitably modified techniques of information retrieval.
The first step of the process is to analyze the sentences and annotate them with syntactic and semantic tags. Despite the fact that documents were provided preprocessed with the GDep parser, we preferred to parse them with a parser that could produce Stanford Dependencies, which provide more refined analysis with respect to standard parsers and whose annotation is closer to represent semantic roles. In particular we will make use of the ability to distinguish apposition.
Besides syntactic information, we looked for some kind of semantic information, in particular grouping syntactic variants and identifying synonyms and hypernyms. 3.1
We explored using the Unsupervised Semantic Parser by [
USP requires the input annotated with Stanford Dependencies [
This process turned out to be too slow to handle large collections. Therefore we
decided to train the DeSR dependency parser [
We applied USP to the Elsevier corpus. We had to overcome a number of problems with the implementation, in particular: ─ the handling of apostrophes, which the parser introduces to denote copy nodes and therefore confuse USP ─ the handling of underscores, whose presence in words also confused USP ─ infinite loops due to the presence of cycles in the dependencies After fixing these problems, we had to partition the Elsevier collection into four smaller subsets because the parser was too slow to handle it as a whole. Unfortunately the clusters that we were able to obtain were not very significant. Here are the top clusters from the first subset: ─ furthermore, however ─ rinse, wash ─ moreover, thus ─ recent, previous ─ feature, manifestation ─ nacl, hepes ─ experimental, procedures ─ the (dep body), the (amod body), the (nn body), the (num body), the (nsubj body), the (advmod body), the (poss body), the (nsubjpass body), … ─ phosphorylation (prep_at s45), phosphorylation (prep_of s45), phosphorylation (amod s45), … Except for the first few, all the remaining ones were cases of alternative parses of the same phrase, e.g. ‘body’ with different dependency relations: ‘amod’, ‘num’ or ‘nsubj’. Therefore the clusters turned out not to be very useful for our purposes. 3 https://sites.google.com/site/desrparser/Announcements/stanforddependencies
Hence as an alternative we resorted to use WordNet [
We exploit the dependency annotations as appositions, to add the apposition at the position of the head. For example, “Abeta” occurs as an apposition for “amyloid beta”: the term “Abeta” is then added as a synonym in correspondence of “beta” and hence it will inherit any relations that the phrase has.
Similarly the expansion of acronyms can be added as variants of terms. 3.3
Syntactic variants are also added to the expanded index. For example to deal with alternative passive forms, when a term has the dependency “nsubj_pass”, to the index is also added the same term with a tag as “dobj”, while the corresponding “agent “ is annotated also as “nsubj”. 3.4
The search engine that we used in our experiments is called DeepSearch and is built
by means of IXE [
DeepSearch provides facilities for passage retrieval, so that it can return individual passages, in our case sentences, matching queries. Scoring is based on a relevance metric for the match within one sentence or within a number of adjacent sentences.
For this task we exploited the capability of IXE of dealing with multiple layers in documents. A layer is an overlay of different terms on the same document. Each layer has its own full-text index and can be queried independently. However the layers can be considered as stacked and querying performed across layers, for instance one can find a word with a given POS by searching for the word in the text layer that has the given POS in the same position in the POS layer. Also proximity and phrase searches that rely on term positions can exploit this.
Besides having multiple layers, a layer can have multiple terms in the same position. This feature will be exploited to deal with variants, like synonyms and hypernyms.
Notice that in among both synonyms and hypernyms we include the lemma itself, hence there is no loss in generality when querying on these layers.
Finally, the column for dependencies is dealt specially in order to enable searching
for terms that are syntactically related. IXE exploits the fact that heads of terms
represents positions, and list of positions are already represented in an inverted index.
Therefore a special posting list with position is created for the heads column. The
technique used for search is based on the Small Adaptive Set Intersection [
The SASI algorithm has been extended to handle dependencies queries, which look for two related terms d and h, where h is the head of d. A special dependency cursor uses three inner cursors, cursor Cd on the posting list of d, cursor Ch for that of h and a cursor Hd on the postings of the heads of d. It first uses Cd and Ch to find matches for d and h. Whenever such a match is found, it checks that the value stored in the posting list of H at the position of d corresponds to the position of h. Having a single posting list for all terms, rather than having one for each term, allows repeating the last step and checking whether there is a transitive dependency. Despite the fact that the posting lists for heads becomes very long, the algorithm is still quite fast since it exploits skip lists to quickly scan such list.
To illustrate what goes into the index, we show here the annotated parse tree for one sentence from one of the reading documents (with slight corrections of parser mistakes) that we will also use as an example for the process of answer selection: form the γ-secretase inhibitor
lemma the γ-secretase inhibitor Semagacestat Semagacestat NN tested test
dep det amod appos nsubj_pass O dobj nmod O
NE O B-protein
O POS DT JJ NN VBN IN NN CD JJ NNS Each column in the table represents a layer, which we represent here vertically for easier readability. Most layers contain a single term. The terms in the last two layers are to be considered as variants; hence they appear in the index as if they had the same position. For eample the sentence will match a query for both form “Semagatestac” and synonym “check”. Notice that the presence of a possibly irrelevant synonym like “run” would not affect any search that does not use it explicitly, which will happen instead if one performs query expansion with synonyms. The DeepSearch query language allows specifying conditions occurring simultaneously at the same position in different layers. Each layer is identified by its name, so for example:
ne:protein matches the term protein in the “ne” (Named Entity) layer.
dep:nsubj matches the term “nsubj” in the “dep” (Dependency Relation) layer. The cooccurrence of these condition at the same position, i.e. looking for a term which is both a protein and the subject of a verb, can be specified using the align operator “|”: ne:protein|dep:nsubj Moreover one can specify the presence of a syntactic dependency, either direct or indirect, between two terms, like in this example:
(ne:protein|dep:nsubj <- lemma:test) Indirect dependencies are useful since quite often terms are connected though intermediate prepositions.
Dependencies can be chained, as in:
(phase <- lemma:trial <- lemma:test) These queries can be tested in our online demo accessible at: http://semawiki.di.unipi.it/alzheimer/
Questions are processed similarly to documents, using the parsers DeSR and then USP, in order to obtain a parse tree with semantic annotations.
A script analyzes the annotated parse tree and generates a query for the enriched index.
Here is an example of the processing of one of the questions in the evaluation: What candidate drug that blocks the γ-secretase is now tested in clinical trials? The sentence is parsed with Stanford dependencies and then expanded with synonyms, hypernyms and syntactic variants, obtaining the following layers (omitting some of the synonyms, hypernyms):
form What candidate drug that blocks
lemma What candidate drug that block the γ-secretase the γ-secretase is now tested in clinical trials ? be now test in clinical trial ?
POS WP NN NN WDT VBZ DT NN VBZ RB VBN IN JJ NNS .
dep nsubj root dobj NMOD rcmod 7 det 10 nsubjpass
dobj 10 auxpass 10 advmod 5 ccomp
NE O O O O O
Hypernym prospect can- person individual someone didate cam- somebody mortal soul politipaigner nomi- cian politico political_leader nee do_drugs drug consume ingest take_in take dose havemedicate medicineagent obstruct freeze computer_memory_unit colobturate im- lection aggregation accumulamobilise tion assemblage shape form auction_block artifact inability cast casting pulley_block obstruction machine obstrucmental_block tion obstructor impediment stymie stop- hide conceal prevent forestall page forget foreclose preclude forbid … jam … O Bpro tei n O O O essay run exam examine evaluate judge take screen exam- submit check core communicaine prove quiz tion communicating experitry trial tryout ment experimentation attempt test examina- effort endeavor try determine tion … check find_out see ascertain
watch learn covering cover … 10 VMOD 13 amod 10 prep_in 8 P
O O clinical O O Among the hypernyms for the word “candidate” appear terms like “politician”. It is fairly clear that if these terms were used for expanding the query, quite confusing results might be retrieved, unless one does some sophisticated kind of word sense disambiguation to discard those terms. In our case though, the last three columns of the analyzed sentence are discarded during query generation.
The query generator creates a basic DeepSearch query, which includes relevant terms and both syntactic and semantic features from the query. A list of clauses is produce which are combined in a Boolean disjunctive query. A few heuristics are used in producing such clauses, for instance the possible answer is not included if it is already present in another clause from the question.
For the above example the query generator produces this base DeepSearch query: candidate OR syn:candidate OR drug OR syn:drug OR γ-secretase OR syn:γ-secretase OR clinical OR syn:clinical OR lemma:candidate OR (ne:protein <- lemma:test) OR syn:drug Five variants of this query are submitted by adding to each, one of the possible multiple answers. The one with the addition of “Semagacestat” returns two results, with the highest score of 19.35. The query with the addition of “LPR1” returns one result with a score of -1.64; the addition of “biochemical” returns 21 results with the best score of 4.24; the addition of “AD” obtains 9 results, with the best score of 9.18; the addition of “PSEN1” returns 21 results, the highest with a score of 5.51. Hence “Semagacestat” is selected as the correct answer, as it is indeed.
The sentence retrieved with the highest score is the one presented in Section 3.4. 4.2
Most Question Answering system perform sophisticated processing on the candidate
answers, in order to determine which one is the most appropriate one. This processing
sometimes involves complex reasoning based on theorem proving techniques [
Our system instead relies only on the ranking provided by the DeepSearch engine. If the engine does not return any answer for a given query, the candidate solution is discarded. When more than one query has answers, the one is chosen whose first answer has the highest rank.
One limitation of the approach is due to the fact we employ a passage retrieval
engine, which splits documents at the sentence level and returns sentences that match
the query. We had planned to apply the anaphora resolution tool [
The metric for the evaluation is c@1 [
Reading Document 4. 1. What effect can be observed when when γ-secretase is blocked?
System: APP-CTF accumulation Correct: APP-CTF accumulation 2. When APH1 genes are overexpressed in MEF KO what happens with the Aβ?
System: They are longer Correct: They are longer 3. In which gene are mutations associated to many cases of early-onset familial forms of Alzheimer's disease?
System: PSEN1 Correct: PSEN1 4. What experimental technique was used specifically to purify the γ-secretase complex?
System: lysate Correct: affinity chromatography 5. What peptide is able to control the expression of the ApoE gene?
System: AICD Correct: AICD 6. Which amino acid is critical for the activity of the PS1 protein?
System: aspartate Correct: aspartate 7. What experimental technique was used to determine the structure of γ-secretase?
System: immunostaining Correct: EM 8. What candidate drug that blocks the γ-secretase is now tested in clinical trials?
System: Semagacestat Correct: Semagacestat 9. What mutation of the PS1 protein causes γ-secretase activity almost to disappear?
System: wild-type Correct: P436Q 10. How many mutations relevant for familial forms of Alzheimer's disease have been detected for the PSEN1 gene?
System: 185 Correct: 185 We briefly investigate the reason for the failures.
For the 8th question the query generated contained the phrase "affinity chromatography", but neither the term “affinity” nor “chromatography” appear in the document.
For question 7, the query with “EM” did not have the highest score. The answer was hard to find, because the relevant sentence was twisted, saying that “protocols for the purification of … γ-secretase … allowed the reconstitution of 3D structures … by EM”. Some form of splitting and rewriting of the sentence might help with cases like this.
Finally, in question 9 “wild-type” prevailed over “P436Q”, because the generated query did not constrain the term “wild-type” to be connected with “mutation”.
These errors appear that could be reduced by improving some aspects of index expansion and query generation.
The approach of index expansion relies on enriching the index with information which is gathered from a linguistic analysis of texts. Differently from traditional approaches to Question Answering, where an abstract separate formal representation of texts is produced and then queried through some logical reasoning process, our approach keeps the information in the text itself.
The idea is to create a highly tangled representation of the sentences where each word is directly connected to others representing both meaning and relations. Instead of keeping the knowledge base separate, the relevant knowledge gets embedded within the text. We can hence use efficient indexing techniques to represent such knowledge and query it very effectively with suitably modified techniques of information retrieval.
The approach proved fairly effective for the Pilot task of Machine Reading for Question Answering. In the experiments we incurred into some limitations of the tools we had planned to use. In particular it is important for the approach to be able to identify alternative forms of expressions, in particular synonyms and hypernyms, which are specific to the domain. We had some success in doing this by using dependency relations produced by a statistical dependency parser. In particular the apposition relation allowed identifying synonyms and the recognition of passive forms allowed normalizing them. We also tried some recent tools for semantic analysis that in principle could have provided additional versions of linguistic variants, more domain specific. This did not work as expected and we had to resort to general linguistic knowledge provided by WordNet. We also did not have time to incorporate our tool for coreference resolution.
We hope that exploiting better or tuned versions of the tools for syntactic and semantic analysis of text, the index expansion approach can provide an effective solution to answer validation in the context of question answering.
Mihai Surdeanu provided us access to a version of the Penn Treebank annotated with basic Stanford Dependencies and assisted us in the training of the DeSR parser.