Question Answering Systems play a significant role to retrieve exact answers for user's specific questions. In answer retrieval process, they employ query expansion methods which play a major role to expand scope of original questions in correct sense. In this paper, we have carried out an extensive survey of few popular web-based open domain Question Answering Systems and critically evaluated their performances on a set of 300 questions from 30 different domains collected from standard resources including TREC to conclude our results. On the basis of findings, we have suggested an efficient query expansion framework that uses multiple ontologies retrieved from semantic web search engine such as Swoogle and combines them with WordNet to disambiguate the context. The proposed approach successfully constructs a conceptual query for user's questions to retrieve relevant answers. We have experimented on a set of 300 questions to judge the effectiveness of the proposed approach.
within a document
A typical Question Answering System, as shown in Fig. 1, takes user’s question in some natural
language as an input. This question is then optionally modified using some query modification
technique (also called query expansion) and output of this modification process is a set of queries
similar in meaning to the original question. These modified questions are fed into knowledge repository
which may be either predefined corpus as in START [
In this paper, we are performing comparative analysis of some popular web-based open-domain
Question Answering Systems using a corpus of 300 questions as described in TREC [
This paper is organized as follows: section 2 provides details of Question Answering Systems and ontologies related work. In section 3, we explain the proposed method of comparative analysis to evaluate performance of selected Question Answering Systems and performance of these Question Answering Systems based on the proposed method. In section 4, we explain the proposed method for query expansion using multiple ontologies and WordNet. We have shown our observations and results in section 5. In the last section, we have stated our conclusion and future directions to build an Intelligent Question Answering System.
There are number of Question Answering Systems like START [
Use of ontologies for query expansion has become popular in recent years. However more focus has
been given on the use of single domain ontology while use of multiple ontologies is quite rare. [
Question Answering System analysis needs inception of an evaluation approach to provide a fair opinion about each Question Answering System and about its performance. To carry out performance evaluation, we have done exhaustive search for Question Answering Systems available on World Wide Web using Google and selected top five systems like START, AnswerBus, PowerAnswer, BrainBoost, and Inferret based on their working methods so that the analysis can cover each possible approaches used by Question Answering Systems. None of these Question Answering Systems are supposed to be specialized in any particular domain and hence domain specific bias has been avoided. We carried out our survey in two components: Question Collection and Answer Evaluation. The detailed description of each component is given below.
In Question Answering System performance analysis, the selection of Question Set draws major
attention. We tried to avoid any bias in the evaluation while preparing the Question Set for evaluation
of Question Answering Systems. We have chosen a set of 300 questions from 30 different domains and
categorized into six groups: Factoid Questions, List Questions, Contextual Questions, Textual Image
Retrieval Questions, Biographical Questions, and Other Complex Questions as described in [
The evaluation criteria do a major job in the completeness of performance analysis. We have divided all the domains into 10 groups and each group is having 3 domains. We came up with evaluation strategy, in which each group will be evaluated by two persons so that correctness of the answers of the questions could be judged properly. We feed each question in listed Question Answering Systems and analyze retrieved answers. The evaluation is performed in two phases. In the first phase of evaluation, we identify whether Question Answering Systems are able to retrieve answers of the asked questions or not and judge the quality of retrieved answers using carefully chosen 11 criteria as shown in table 1. These criteria are based on two parameters which are as follows: How easily answers are available to the user? How easily they are understood by an average user?
To measure criteria’s importance, we introduce a weighted scheme where weight can be assigned
from [
In the second phase of evaluation, modified form of the same set of questions were asked from Question Answering Systems. We have modified original questions using Changing keywords positions, Word Substitution, Changing the voice, Sentence formatting, Expansion/Contraction of acronyms, and Factual Rephrasing. The aim of the modification of the question was to test whether or not the Question Answering Systems were able to answer the rephrased questions as different users generally use varied format to ask the same thing. Since these questions were already answered by them in the first phase of evaluation, it is obvious that answers of these questions were already present in the knowledge base of the Question Answering Systems.
Correct answer displayed on one of the link but it requires small interpretation. (e.g. interpreting proper nouns) Correct answer in expected format found in the first paragraph of the web page pointed by one of the top five links Correct answer in expected format found in the first paragraph of the web page pointed by one of the next top 5 links Correct answer in expected format in one of the documents pointed by any of the top 10 links and the document requires careful study Correct answer in expected format found in the first paragraph of the web page pointed by one of the links ranking 11 and more Correct answer found in one of the links with depth 2 or more than 2 Answer found in first paragraph of web pages pointed by one of the links ranking 1-10 but finding answers require some manual interpretation of sentences Answer found in web pages pointed by one of the links ranking 1-10 but finding answers require some manual interpretation of sentences Correct answer in expected format found in the first paragraph of the web pages pointed by one of the links ranking 11 and more and finding answers require some manual interpretation of sentences No answer at all Wt The results of the first phase of evaluation are shown in column 2 of table 2 where the highest percentage match of correct answers is retrieved by Inferret which uses statistical approaches to retrieve answers. These results show that recall is more important than precision in the case of very large corpus like World Wide Web. AnswerBus and PowerAnswer have performed consistently well in TREC evaluations because of sophisticated NLP techniques for query and document processing. In case of large knowledge repository like World Wide Web, overall performance of the Question Answering System is influenced more by recall than by precision. This explains the lower performance of NLP-based Question Answering Systems such as PowerAnswer as compared to Inferret. Still their performance is much higher than BrainBoost and START mainly due to two factors: Using World Wide Web as knowledge repository and very sophisticated processing of the retrieved documents resulting into the high precision. The BrainBoost is on 4th position because it first tries to find the answer from the set of similar questions prepared by answers.com and it searches the web up to a certain degree. START retrieves the lowest percentage match of correct answers because it does not perform search on the entire web rather searches in its own created knowledge database. The next important question is judging the quality of retrieved answers. Are Question Answering Systems able to retrieve correct answers in a simple manner? Therefore, we performed evaluation of the answers in terms of quality using criteria’s discussed in section 3.2. In table 2, N1 represents the no. of retrieved answers which are having weight 1.0 while P shows the precision value of answers having weight 1.0 out of total number of correct answers (N).
In table 3, we show that the quality of correct answers retrieved by START is highest. START is having 139 correct answers of weight 1.0 (out of 144 correct answers) while Inferret is on the bottom because quality of answers was not appreciated by the user. We also classified the retrieved answers, based on the weighted scheme, into three categories that is good, average, and poor to get a better picture of each Question Answering Systems’ performance. We consider an answer with score 1-0.8 as “good”, from 0.7-0.5 as “average” , and “poor” (less than 0.5) as shown in table 3.
In the second phase of evaluation, we have modified 300 original questions and retrieved results. We found a steep degradation in the retrieval of correct answers. In table 4, we show the correct retrieval data for each Question Answering System when the user asked the same question in a different format. We are calculating decline in recall percentage using following formula:
F = (N-N2)*100 / N .
(1) where N2 represents the number of correct answers retrieved for modified questions. Table 4 shows that that BrainBoost is very sensitive in the identification of questions and very much dependent on question language while Inferret shows the lowest decline which can be mainly attributed to its efficient usage of large corpus like World Wide Web.
The results of table 2, table 3, and table 4 clearly prove the inefficiency of the existing Question Answering Systems in question interpretation and retrieval of correct answers. Their retrieval of correct answers is very much dependent on “question form” and not on the meaning of the question. We are 20 18 16 14 12 10 8 6 4 2 0
START AnswerBus BrainBoost PowerAnswer Inferret showing this in fig. 2 by considering 20 questions which are modified by changing parameters discussed in section 3.2 for each Question Answering Systems. In case of changing keyword positions, AnswerBus and Inferret were remained mostly unaffected with a retrieval result of result of 80% and 85% while performance of START, BrainBoost, PowerAnswer went down drastically with a retrieval result of 50%, 45%, and 60% respectively. This is very much obvious in the given example question like "How far is the Mars from the Sun?" where the question is answered by BrainBoost but it failed to answer the modified question like "How far is the Sun from the Mars?”. Even very sophisticated Question Answering Systems like START and PowerAnswer were affected by this kind of question modification.
Changing keyword positions
Word Sentence Factual substitution formatting rephrasing
In case of factual rephrasing, we observed a sudden decline in the performance of each Question Answering System. Their performance went down from 30% to 55% with Inferret and START respectively. These modifications imply about the reasoning capabilities of Question Answering Systems, as the question "How far is the Mars from the Earth?" was modified into “How far is the Mars from our planet?” with the hope that Question Answering System will modify “our planet” with “Earth”. With these observations, this is very much clear that existing Question Answering Systems need an efficient methodology to interpret asked question using discussed parameters to improve user satisfaction. 4
Ontologies and WordNet are having rich information about domains and semantic relations between concepts. Query expansion methods based on only selected relations in WordNet have resulted into degradation of Question Answering Systems’ performance. On the other hand, if we use all the relations in WordNet in an uncontrolled manner then we get more number of semantically related words which forms a large number of modified queries against user’s single question. So, this method is not viable because of the requirement of more computational resources.
We are proposing multiple ontologies and WordNet based query expansion method. This method takes user question as an input and extracts key concepts from the question and then automatically finds the most relevant senses for key concepts of the question using WordNet. To disambiguate the correct sense, the method uses multiple domain ontologies retrieved from Swoogle semantic search engine. When key concept(s) from the user’s question is fed into Swoogle then it returns ontology classes describing the concept(s). In the proposed method, we compute semantic distance of the key concepts from retrieved ontology class, super class, and its subclasses and consider the class with the lowest semantic distance for the further processing. The complete proposed algorithm is given as follows:
Step 1: Let T be a set of quadruples and defined as T=<C, O, W, R>, where C denotes concept in user’s question, O represents ontology for the concept C, W represents weight of an ontology O, and R represents one of the semantic relations retrieved from WordNet. Initially, T is empty. Step 2: User enters a query Q.
Step 3: Extract key concepts C1, C2 ...Ck from Q.
Step 4: User assigns W1, W2 ... Wk weights to the concepts C1, C2 ...Ck on the scale of 1-10. The concepts with higher weights are considered as important concepts.
Step 5: Search Swoogle for the combination of concepts using term dropping strategy. Query for the Swoogle is fed into Conjunctive Normal Form. All ontologies describing a concept combination are put into one group. Let us assume ‘n’ ontology groups defined as OG1, OG2, ...., OGn.
Step 6: Let WNc1, WNc2,…,WNck be the domain set in WordNet for concepts C1,C2 ..Ck. .Elements in WNci are denoted by couple (S, R) where S is synonymous set for concept Ci and R is relation in WordNet that connects Ci to S.
Step 7: for (i=n; i>0; i--) do following for each ontology of OGi group.
T= T (<C1, x, WOGi, R1>, x(Oij WNc1) ) (<C2, x, WOGi, R2 >, x(Oij WNc2)) … (<Ck, x, WOGi, Rk>, x(Oij WNck)) (where Oij is jth ontologies of ontology group OGi ) Step 8: If T is empty for each Ci add one sense from all relations available in WordNet to T. We select most frequently occurring sense of the word and assign zero weight to the ontologies. Step 9: QE= (C1 OR O11 OR O12…OR O1m) AND (C2 OR O21 OR O22…OR O2n) AND …….(Ck OR Ok1 OR Ok2…OR Okr) where Oij is the common ontology for concept Ci found in previous steps.
We explain proposed algorithm with the help of an example. Let us consider a query in the form of question i.e. “What is Jupiter's atmosphere made of?". The key concepts found in this question are written as C1 = “Jupiter”, C2 = “atmosphere”, and C3 = “made”. The key concepts in the question are obtained by removing the wh-words and the stop words. Our system analyses the question and derives the expected answer type before actual removing of the wh-words and the stop words takes place. The key concepts are then presented to the user and the user assigns weights to each concept as W1= 9, W2=9, W3=3 respectively. We do start searching of relevant ontologies from Swoogle using term dropping strategy. For the example query, the search strings are “Jupiter AND atmosphere AND made”, “Jupiter AND (atmosphere OR made)” , “(Jupiter OR made) AND atmosphere” , “(Jupiter OR atmosphere) AND made” , “Jupiter”, “atmosphere”, and “made”. Swoogle provides many advanced meta-tags for specific searches. We are using meta-tag called “desc: term1” to retrieve ontologies which are having “term1” in the description of the document, generally in the annotations. We are passing queries to Swoogle in Conjunctive Normal Form to retrieve ontologies which are relating given concepts in some meaningful way. Swoogle uses “AND” as a default logical operator. In step 7, we find common concepts between ontology group and WordNet group like in 5th query, concept planet is common in both OG5 and WNJupiter. Hence quadruple for this query will be < Jupiter, Planet, 9, hypernym> and will be added in T. We do step 7 and step 8 for all queries and get final set T = {<atmosphere, air, 18, synonym> <Jupiter, Planetary Object, 18, hypernym>, <Jupiter, Planet, 9, hypernym > <atmosphere, weather, 9, hypernym>, <make, constitute, 9, synonym>}. Therefore, final expanded query is defined as QE = [(Jupiter OR Planet OR Planetary Object) AND (atmosphere OR air OR weather) AND (make OR constitute)]. In the next section, we will run our proposed algorithm for large number of questions and will also compare results with existing Question Answering Systems.
To judge the accuracy of proposed query expansion method, we have considered a set of 300 questions collected from standard sources like TREC etc. (same set as discussed in Sec 1) which are covering almost 30 different domains. We are representing top 20 questions and their corresponding expanded queries in table 5. (Jupiter OR Planet OR Planetary Object) AND ( atmosphere OR air OR weather) AND ( make OR constitute) ( reason OR cause ) ( sky OR rainbow OR cloud OR lightning) AND ( blue OR sky-blue) AND color planet AND ( least OR smallest OR minimum) AND surface AND (acraeapital OR “state capital” OR means OR Centre OR “Graphic symbol”) AND Susquehanna AND river (far OR distant) AND (Mars OR “Red Planet”) AND our AND (Planet OR “terrestrial planet”) (famous OR celebrated OR known OR notable) AND ( invention OR creativity OR creativeness OR “creative thinking” OR “ creating by mental act”) AND ( Marconi OR Guglielmo Marconi) (“Life Expectancy” OR “Life Expectancy at Birth” ) AND average AND woman AND ( Nigeria OR Lagos OR Zaria OR “Yerwa Maiduguri” OR Niger OR Africa ) ( Countries OR country OR land) AND ( border OR “has border” OR “borders on”) AND( India OR Indian) ( deepest OR deep) AND ( sea OR Ocean) (continent OR subcontinent OR landmass OR Asian OR African) AND ( India OR Indian) (state OR province) AND ( longest OR long OR length) AND coastline AND Atlantic AND Ocean Fraction AND (Ozone OR “Ozone layer” OR stratosphere Or oxygen) AND layer AND depleted Year AND ( Beethoven OR Ludwig van Beethoven OR music OR composer) AND( born OR “born in”) Composer And ( opera OR “comic opera” OR “opera bouffe” OR bouffe OR “opera comique” ) AND semiramide (music OR Bach) AND (Debussy OR Claude Debussey, Claude Achille Debussy) AND (composed OR compose OR composer OR “composed for ” OR “ is composed of” OR “ composed for” OR “ music composed by”) Year AND (“Arundhati Roy” OR Arundhati) AND ( get OR receive) AND Booker AND( prize OR award) (Seventh OR 7th ) AND president AND (India OR Indian)
expectancy” OR “ life expectancy at birth”) (Taiwan OR Taiwanese OR Taipei OR “South China sea”) AND ( flag OR “national flag”) AND ( look OR appear) AND (like OR “likes of”) Famous AND communist AND leader AND (died OR “died in year” OR death) AND Mexico AND ( city OR Leon OR “Acapulco de
Juarez” OR Tepic OR Culiacan OR Matamoros OR “Tuxtla
We fed 300 original questions and their expanded queries in Google separately and evaluated top 10 retrieved results for each original question as well as for each expanded query. With the set of original questions, we find satisfactory answers for 258 out of 300 questions while with the set of expanded queries satisfactory answers retrieved for 276 out of 300 questions. The proposed approach has retrieved 92% answers correctly which is an overall improvement of about 8% in comparison with answers retrieved for existing set of original questions.
To measure the performance of proposed approach, we have experimented with some existing popular web-based automatic Question Answering Systems like START, AnswerBus, BrainBoost, and Inferret. We reformulated expanded queries as per the Question Answering System specific format and fed them in all Question Answering Systems. The overall performance of each Question Answering System shows a significant increase in retrieving correct answers. The performance bar chart has been shown in figure 3 where BrainBoost and Inferret indicate an improvement of 25%, START exhibits an improvement of 33% while AnswerBus records maximum improvement of 44%. The overall average improvement is 31.75% on Question Answering Systems which is very significant as these systems are already using very sophisticated information retrieval techniques to retrieve correct answers. On the basis of experimented results, we can say that proposed approach is working reasonably quite well.
In this paper, we have evaluated the performance of START [
Further, we have presented that how semantic web and WordNet can be effectively utilized for World Wide Web based Question Answering Systems. We have concluded on the basis of experimented results that combination of semantic web with vast and exhaustive lexical resources like WordNet can greatly improve the performance of the Question Answering Systems. We have proposed query expansion framework using ontologies and WordNet to expand the original question’s scope conceptually. Similarly, this can be extended for other phases of Question Answering System. In future, we are intending to develop an efficient content based Question Answering System using ontologies and WordNet.