This year we participated in the English monolingual paragraph selection task at ResPubliQA 2010. Our general strategy is to find the supporting word context from query and candidate passages during the paragraph selection. We use the techniques of state-of-the art publicly available question answering systems, i.e. Open Ephyra and JIRS, and the random projection implementation in the Semantic Vectors package to evaluate the word context. To strengthen the paragraph selection, besides the context evaluation, we also use n-gram overlapping and textual containment. Our approach has a c@1 measure of 0.73 for our pattern-based context configuration and 0.64 for our n-gram-based context configuration.
Question Answering (QA) is a specific form of information retrieval (IR) that
seeks to produce an exact answer given a natural language question. An automated
QA system tries to retrieve explicit answers in the form of a single answer or snippets
of text rather than a whole document or set of documents. The main techniques that
mostly have been used in current QA research are semantic analysis using semantic
role labeling [
This year’s ResPubliQA1 evaluation campaign is a continuation from last year
which tries to evaluate QA performance in a specific context, i.e. the legal domain
[
selection strategy which validates the passage retrieval mechanisms from two
stateof-the-art publicly available QA systems, Open Ephyra [1] and JIRS [
Our general hypothesis in this experiment is that, in a specific domain, the context of a question is near to the context of candidate paragraphs that are suggested during the passage retrieval. This hypothesis is a generalization of the distributional hypothesis from the word space methodology, which says that words with similar meaning tend to occur in similar contexts [3]. We took the publicly available random projection implementation in Semantic Vectors [4] as the context supporting system of our approach. Our observation during this experiment will be: how to employ word context to support passage retrieval performed by language model and n‐gram approaches. To answer this research question, our approach tries to boost the influence of word context during paragraph selection.
In this paper we describe the approach of our context supporting strategy and the results obtained in the monolingual English paragraph selection task. The rest of the paper is structured as follows: in section 2 we describe in general the related systems whose techniques were used in our approach, namely Open Ephyra, JIRS and Semantic Vectors. The details of our approach will be covered in section 3. Section 4 will show some results and analysis in the paragraph selection task of this year’s ResPubliQA. Finally, some conclusions and suggestions for future work will be given in section 5. 2
Open Ephyra2 and JIRS3 are two QA systems which offer comprehensive pipelines. They base their passage retrieval strategies on, respectively, language models and ngram structures of the passages. Both strategies rely on the probability and sequences of adjacent words from query-passage pairs, but do not really observe the word in context. Alternatively, Semantic Vectors4 offers the possibility to observe the word context in a domain by implementing the word space methodology.
Open Ephyra (OE) uses a pattern learning approach to categorize questions [
There are two main steps for the pattern-learning approach in OE: 1. The first is to learn the question patterns from question templates according to each question type. The aim of this step is to interpret the questions and 2 http://www.ephyra.info 3 http://sourceforge.net/projects/jirs/ 4 http://semanticvectors.googlecode.com/ transform them into queries. The question templates need to be manually developed according to various interrogative sentences that are independent for each natural language. 2. The second step is to learn the answer patterns from question-answer pairs. The aim of this second step is to extract answer candidates from relevant document snippets and to rank them.
In our experiment, we only used the first step to develop an appropriate set of question patterns for each question type according to the training data from last year’s ResPubliQA. A recognized question pattern for a given question is essential in order to extract important keywords (i.e. target, context and property) for querying purposes. The retrieval phase in OE is done within the Indri search engine that searches for passages in documents - based on the recognized query keywords - which could contain the answer candidates.
The question answering module in JIRS is an extension of the JIRS passage retrieval
system as described in [
JIRS is based on the idea that in a large document collection, an n-gram related
with a question will be found in the collection at least once [
Semantic Vectors is an open source package that can be used to build context vectors of word concepts in a specific domain. It implements word space methodology [3] by applying random projections of words in the document collection. Words in Semantic Vectors are represented as vectors in a high dimensional space, where words or documents that have related meanings are in close proximity. By applying random projection, the computational resources that are usually required for computing semantic similarity, as for instance in Latent Semantic Analysis (LSA), can be minimized.
The random projection strategy can be summarized in two main steps [
For each document, make an N-dimensional ternary index vector by placing a small number, k, of -1’s and +1’s (e.g., ten of each; k = 10) at random among the N-dimensional index vector, the rest will be 0’s. 2. Step 2: Build the document vectors
Scan through the documents. Every time the word appears, add the index vector to row w of matrix G, where G is a (M x N) matrix, M is the number of words in the collection and N is the number of reduced columns.
The Semantic Vectors package uses the Apache Lucene API5 to create a word space model from a term document matrix, using random projection to perform dimensionality reduction.
3.1
For the data preparation, from the JRC-ACQUIS6 and EUROPARL7 corpus we have created a passage index based on the paragraph segmentations. In total we have around 1.5 million passages. We created separate indexes for each information retrieval system that we used in our approach, namely Indri, JIRS and Lucene.
Indri is a search engine that is specially designed for passage retrieval [2] such as
JIRS. The difference between them lies in the retrieval model. Indri’s retrieval model
is based on a combination of language modeling and inference network retrieval
frameworks [
The Lucene index will be used by the Semantic Vectors package to form the terms and document vectors [4]. In our approach we choose the value of 2000 for the number of dimensions. This value is the reduced columns for documents projection that is considered as the word space. 3.2
The high level architecture of our approach can be seen in Fig. 1. This architecture is based on the general framework of question answering systems. The first component is the question analysis, which normalizes the question (i.e. removes the question word, stop-words, and performs word stemming), and determines the question type. We used manually developed question patterns to determine the question types.
After the question analysis step, the system delivers the normalized question to a
query generator based on the techniques that were used in OE [
The main idea during paragraph selection is to evaluate whether supporting word context can be found among the candidate passages. It checks first the word context
6 http://wt.jrc.it/lt/Acquis/
7 http://www.europarl.europa.eu/
that is present at the top-1 retrieval of the candidate passages. This is done by
comparing candidate passages from Indri or JIRS with the one retrieved by Semantic
Vectors. If the two systems that were compared returned the same document at the
top-1 retrieval, then we believe that supporting word(s) context was found in the
passages that were compared, or in other words the documents “shared” the same
context. Otherwise, the paragraph selection will be determined by applying n-gram
overlapping, textual containment [
From this initial experiment, we can see that our proposed approach, as described in previous sub-section, outperformed the baseline accuracy of Indri and JIRS by 6.8% and 6.2% respectively. The graphical interpretation from the above experimental result is presented in Fig. 2.
For our first submitted run, we combined the technique of pattern-based question
answering, as proposed in [
ANSWER_TEMPLATE: Opinion: <PO> what (a|the)? <CO> (position|opinion|feelings|ideas) (with respect to) <TO>
The detailed strategy of the paragraph selection phase can be seen in Listing 2. In this scenario, the supporting context (the first step in listing 2), is used to select passages that we believed ‘share’ the same context, and hence they can be used confidently as the final answer. If the result of Indri and Semantic Vectors did not return the same passage, we evaluate the top-5 of Indri results by applying n-gram overlapping (step 2.a) If n-gram overlapping is unsuccessful, i.e. all passages have the same amount of n-gram overlapping, we use a textual containment strategy to calculate the best proportion of query terms that can be found in the passages (step 2.b), and at the same time try to find a named-entity that is expected to be present in the passages (step 2.c). ‘No answer’ will be returned if there is no passage that meets the conditions in the strategy (step 3). Note that by using containment, passages whose text is too long will be discarded. This step is used essentially to anticipate the definition question type, which usually only has one or two terms from the query that can be found back in the candidate passages. 1. Compare the retrieved passages by Indri and Semantic Vectors. If the top-1 returned document is the same, then return the passage offered by Indri as final answer. 2. Otherwise: a. Get top-5 retrieved passages from Indri retrieval, count the query-passage n-gram overlapping (unigram, bigram and trigram) from each passage, return passage with the biggest total n-gram overlapping as final answer. b. If the total count of n-gram overlapping of top-5 passages is all the same, then compare the query supporting text in each Indri retrieved passages (textual containment). Sort the passages by the best containment as candidate answers. c. If the expected answer type is about facts (time, place, number, organization), definition (about a term), or opinion (a person name or organization), check the result from step 2.a. If a named-entity can be found in the first passage, returns it as final answer. Otherwise, try to find named-entity in the lower rankings. 3. If there is no passage found that confirm steps 1, or 2, returns No Answer (NOA).
Listing 2 Detail Strategy of Scenario 1 (Pattern-based – Context Support) 0008 0001 0026 0027
p_id=13 doc_id=EP_CRE20090507-EN_cl.xml
p_id=45 doc_id=EP_CRE20091008-EN_cl.xml
p_id=13 doc_id=jrc32002D0268 -en.xml p_id=164 doc_id=EP_TA20090114-EN_cl.xml
Top-1 SV Retrieval EP_CRE20090507
EN_cl.xml
As an example from Table 2, the Quest. # 0008 (Name a purpose of MEDIA Mundus), Indri and SV have both retrieved the same top-1 document (EP_CRE20090507-EN_cl.xml). According to our strategy, the system should take step 1 in Listing 2 as the final decision for the paragraph returned.
In another example from Table 2, the Quest. # 0027 (Define children footwear),
Indri and SV have retrieved different documents. The next step is to determine the
best n-gram overlapping (step 2.a) between the top-5 Indri retrieved passages and the
question terms (children footwear). Since all of the top-5 Indri retrieved passages
have only one term in common with the question (i.e. the term children), the system
should now compute the value of textual containment [
For our second submitted run, we combined the results of JIRS and Semantic Vectors to retrieve passages which could contain answer candidates. The detailed strategy of the paragraph selection phase can be seen in Listing 3. 1. Compare the retrieved passages from JIRS and Semantic Vectors. If the top-1 returned document is the same, then return the passage offered by JIRS as the final answer. 2. Otherwise: a. Get the top-5 passages from JIRS retrieval, count the querypassage n-gram overlapping (unigram, bigram and trigram) from each passage, return passage with the biggest total n-gram overlapping as the final answer.
b. If all of the top-5 JIRS passages have the same total count of n-gram, then count the n-gram overlapping (unigram, bigram and trigram) of the query in each paragraph of the retrieved top-1 Semantic Vector document, return passage with the biggest n-gram overlapping as the final answer. c. If there is more than one paragraph from the SV top-1 document have the same total count of n-gram then return paragraph with the best textual containment as the final answer.
The first paragraph selection strategy is to compare the top-1 retrieved passage from JIRS and the one retrieved by Semantic Vectors. If both retrieval systems suggested the same document, then we took the passage returned by JIRS as the final answer. Otherwise, we followed the same strategy as scenario 1 to select the best paragraph. But now, we used the top-1 suggested document from Semantic Vectors as the final paragraph decision. Table 3 gives some examples of the system’s decision according to this second scenario for the test questions from ResPubliQA 2010. The explanation for Table 3 is similar to the explanation for Table 2 in the previous subsection.
In Scenario 2, there was no strategy developed for the NOA-answer. The reason for this is considering the final decision rule that selects paragraphs from only one document, i.e. the Semantic Vectors top-1 retrieved document - with no textual containment threshold - and thus, there must be one paragraph returned as the final answer.
JIRS-Semantic Vectors (uiir102PSenen). These two configurations differ in the way passages are refined during the paragraph selection phase (cf. Section 3).
If we observe the accuracy of each question type in detail (Table 5), the ‘Definition’ question type gives the lowest accuracy in each configuration, i.e. 0.43 (Scenario 1) and 0.37 (Scenario 2). This result indicates that the ‘Definition’ question type is the most difficult one to be answered, and our strategy is not suitable to answer this type of question properly. The best accuracy in both scenarios is achieved for the ‘Reason-Purpose’ question type, i.e. 0.87 (Scenario 1) and 0.77 (Scenario 2).
It is interesting to explore the accuracy of the ‘Procedure’ question type, which achieved 0.58 accuracy in Scenario 1, and 0.68 in Scenario 2. Most of the question types have question patterns and question terms that are almost identical from one question to another; see for example the ‘Opinion’ question pattern in Listing 1. This is also one of the reasons that Scenario 1 performed better than Scenario 2, except for the ‘Procedure’ question type that has richer patterns and more answer variations.
RIGHT ANS. WRONG ANS.
Sce. 1 Sce. 2 Sce. 1 Sce. 2 13 12 3 3 1 2 6 2 3 4 1 0 3 4 4 2 4 2 0 3 In this paper we have described our context supporting approach in passage retrieval question answering for the English monolingual task. Our experimental results showed that in a specific domain, context supporting - as a ‘second-opinion’ decision system by using random document projection - can improve the performance of the passage retrieval (paragraph selection) component in a QA system.
It would be interesting to conduct a further study in how to relate word context in broader domains, to support open-domain and multilingual question answering. It would also be interesting to investigate the value of reduced dimensionality from a document collection that formed the word space during retrieval. In this experiment we have only used the number of dimensions of 2000, due to technical restrictions. Perhaps, if the value of the dimension is varied, the context supporting strategy could perform better.