This article outlines the participation of the Documents and Linguistic Technology (DLT) Group in the Cross Language French-English Question Answering Task of the Cross Language Evaluation Forum (CLEF). Our aim was to make an initial study of cross language question answering (QA) by adapting the system built for monolingual English QA for the Text REtrieval Conference (TREC) in 2002 (Sutcliffe, 2003). Firstly, therefore, we outline the architecture of the TREC system which formed the basis of the work reported on here. Secondly, the many changes made to allow cross-language QA are described. Thirdly, the runs performed are presented together with the results we obtained. Finally, conclusions are drawn based on our findings.
In this section we summarise the structure of the DLT system used at TREC which formed the starting point for CLEF. Changes subsequently made are documented in the next section. Overall flow of control was as follows. Firstly, we identified the query type and hence decided upon the related named entities for which we would be searching. Secondly, we parsed the 50 TOPDOCS text files, dividing them into textual units using the markup. These files were supplied by TREC and were generated by their own retrieval system in response to the input query. Thirdly, we searched for instances of the named entities in the textual units and marked any which were found. Fourthly, we identified the winning instance using one of two possible strategies: highest-scoring or mostfrequent. These stages are now dealt with in more detail.
We studied questions of each type and developed simple keyword-based heuristics to recognise them. This crude approach was suprisingly effective. In TREC 2002 425 of the 500 queries were correctly classified.
Each document within the TOPDOCS file was divided into a series of segments corresponding to a short passage of text. First, text within a HEADLINE tag was extracted. Second, text within a TEXT tag was extracted and divided up into separate Ps. Finally a P was divided wherever three contiguous blanks were found. This last stage was to approximate sentence recognition. the resulting textual units were used in subsequent processing. 172 Quelle est la capitale de la Tchétchénie? 57 Quel est l'état indien qui a le plus grand
nombre d'habitants? 169 Dans quel pays le Mont Kilimanjaro se
trouve-t-il? 45 Où se trouve l'écosystème artificiel
appelé "Biosphère 2"? 76 Combien de salles de classe a-t-on construites en 1976 dans le district scolaire
de Wilsona? 164 Quelle est la longueur de la côte de la Which is the length of the coast of bay of
baie de Santa Monica? Santa Monica? 148 Qui est l'ambassadeur des États-Unis en Who is the ambassador of the United States
Suisse? in Switzerland? 178 Quand Shapour Bakhtiar est-il mort? When Shapour Bakhtiar did he die? 160 Donner le nom d'un philosophe To give the name of a German philosopher.
allemand.
Which is the capital of Tchétchénie? Which is the Indian state which has the
greatest number of inhabitants? In which country the Kilimanjaro Mount is
it? Where is the artificial ecosystem called
"Biosphere 2"? How many classrooms did one build in 1976 in the school district of Wilsona?
The type of question identified in the first step determined the type of named entity or entities to be searched for, as is standard practice in QA systems. Each segment identified as above was therefore inspected and all instances of appropriate named entities were tagged.
Two methods were used: highest_scoring and most_frequent. In the first, we returned the named entity occurring in a textual unit which matched the keywords in the query best, chosen from any of the 50 TOPDOCS documents. In the second, we returned the named entity which most frequently occurred in the vicinity of query keywords, observed across all occurrences of the entity in the 50 TOPDOCS documents. Both strategies were unsophisticated but sometimes one or other of them can perform well on a particular query type. In the next section we explain how the above architecture was adapted for CLEF.
The CLEF system had many similarities with the TREC one but also differed in a number of important respects. Flow of control was as follows. Firstly, the type of the French input query was identified along with the appropriate named entities. Secondly, the query was translated and processed in order to produce a search expression for retrieval. Thirdly, the search expression was submitted to a text retrieval engine yielding a set of documents. Fourthly, appropriate named entities were recognised in these. Fifthly, the winning named entity was identified.
As the input query was in French, our existing type identifier had to be re-written. This was undertaken by the third author as part of a separate project in which a French monolingual version of our TREC system was developed (O'Gorman, 2003). The first step was removal of diacriticals from the query, replacing each by the same letter minus the diacritical (e.g. ‘á’ becomes ‘a’ etc.). After this the query was converted to lower-case. Finally, simple keyword combinations were used to identify the query type. 19 query types were used and some of these can be seen in Table 1 together with an example of each.
The next stage was translation of the French query into English. This was accomplished by submitting it to Google with original capitalisation and diacriticals (Google Translation, 2003). The result was then tokenised and stopwords were removed from any material not in double quotes.
Unlike in TREC, retrieval at CLEF was accomplished by indexing the documents ourselves. The document collection comprised 113,005 LA Times articles from 1994, 425MB in all. The search engine used was DTSearch (2000). Documents were indexed using the file segmentation option working with ‘<DOC>’ tags. The engine supports a number of search operators including distance between words (e.g. Word1 w/1 Word2 meaning Word1 must occur within 1 word of Word2) and boolean operators AND and OR. The next step therefore was to produce a search query on the basis of the terms identified in the previous step as follows: · · · ·
A 'w/1'connector was inserted between two capitalised words. It specifies that the second word must occur within one word from the first (in either direction). This feature was helpful because French to English translation often reverses the order of proper nouns.
Double quotes were removed and the string of text within the double quotes was retained, but only if the first word after the double quote was untranslated. This was verified by checking the membership of the word in the original query list in French. Untranslated quotations were searched for exactly.
An AND connector was inserted between all other terms.
If the query included the atom 'did ',it was removed. If the rest of the query contained a verb which was among the list of verbs common to TREC questions, the verb was replaced with its past tense form. For example, ‘die’ in questions like ‘How did X die?’ was replaced with ‘died’.
The query was then submitted to the engine and the first n DOCs returned were used in subsequent processing. In most cases n was 10 while in one experiment it was 20. Before the named entity recognition stage, each DOC was subdivided into separate P elements where present. These were then further subdivided wherever three contiguous blanks were found – a process which approximates to sentence recognition.
The set of English named entities used and the methods for recognising them in CLEF were very similar to TREC (Sutcliffe, 2003). Only two were added, general_name and planet. Type general_name recognises any sequence of up to five capitalised words interspersed by optional prepositions. It was inspired by Clarke et al. (2003) and is used in cases where the question type can not be determined. Planet uses a simple list of the eight planets.
As previously, two forms of answer selection were used, highest-scoring and most-frequent. In the highestscoring method the named-entity instance is selected which occurs in the vicinity of the maximum number of keywords taken from the translated query, across all document passages. In the most-frequent strategy the namedentity occurring most frequently overall across all passages is chosen.
We submitted two runs. These only differed in respect of the answer selection strategy. The first run used the highest-scoring strategy working with the best 10 documents returned by the retrieval system. The second run adopted the most-frequent approach, also using the best 10 documents. The query classification module was the same for both runs. what_city what_state what_country what_continent where how_many3 distance speed temp population who when colour what_river what_water_m ass what_mountain what_mountain _range what_planet unknown Totals
Results are summarised by query type in Table 2. In respect of query classification it shows for each query type the number of queries assigned to that type which were correctly categorised along with the number incorrectly categorised. The overall rate of success was 79.5% which is broadly comparable to the 85% achieved in TREC. However, 90 queries were classified as unknown, i.e. nearly half of the set of 200. 58 of these were correctly classified. This shows the need to add further query types to the system in order to reduce unknown queries to a minimum. In addition various phrases which were frequently used in queries were not anticipated. Examples include ‘à quelle époque’ and ‘à quel moment’ , both of which were classified wrongly as unknown rather than when.
The performance of question answering in Run 1 can be summarised as follows. Out of the 159 queries classified correctly, 21 were answered correctly. Out of the remaining 41 queries classified incorrectly a further 2 were answered correctly. Overall performance was thus 23 / 200 i.e. 11.5%. Results for Run 2 were very similar. 21 of the 159 queries were answered correctly along with one of the 41 queries giving a total of 22 / 200 i.e. 11%. In both runs 117 questions were answered NIL. We discuss these results below.
We used a Dell PC running Windows 2000 and having 256 Mb RAM. The whole system was written Quintus Prolog Release 3.4.
Overall performance of our system was modest at 11.5% in Run 1 but this was not worse than that in TREC 2002 where the score was 10% in Run 1. The best performance was on queries of type what_city in which the system scored 5 / 9 i.e. 56%. Our system was simple, re-used as many components as possible and was constructed in a short timeframe. Our limited aim was to identify the key issues in cross-language QA and in this we were successful. We now consider the components of the system in turn before discussing our findings regarding the project in general.
Query categorisation was not a success with 90 queries classified as unknown. Performance on the remainder was 79.5%. To improve this, further examples need to be examined and appropriate keywords and phrases added. It is likely that our simple approach can give a better result even with such simple changes and without the need for complex analysis. Turning to query formulation and document retrieval, this was the first time we had used our own retireval engine – for TREC we used the TOPDOCS and O’ Gorman (2003) searched the web with Google. The query formulation using boolean operators and exact searches for untranslated quotations achieved much higher precision than was possible with the TOPDOCS where the PRISE engine no doubt uses a vector space type algorithm. On the other hand such an approach can lead to no documents being returned in the first instance, whereupon a query relaxation technique must be adopted in order to re-submit a slightly more general query. Unfortunately we had no such strategy and this was the main reason why so many NIL results were returned (117 / 200 queries, i.e. 58.5% overall).
For named entity recognition essentially the same system was used as for TREC. This performs quite well and its
main limitation is missing entity types which could readily be added. The main addition here was the entity
general_name. In just two cases its use for processing queries of unknown type resulted in correct answers. For
Query 106 ‘Of which political party Rudolf Scharping is member?’ Run 1 returned the correct answer ‘Social
Democratic’ . Run 2 returned the answer ‘Kohl’ . Run 2 returned the correct answer to Query 160 ‘To give the
name of a German philosopher’ : ‘Habermas’ . Run 1 returned the answer ‘Western’ . Finally, in at least one case
where the answer was wrong it was not nonsensical – Query 8 ‘Which largest is the exporting country of pétrole
gross?’ (system’ s answer: ‘Kuwait’ ). The last component of the system is answer selection. The two strategies
used (highest-scoring and most frequent) were the same as in TREC and did not perform well. No answer
patterns such as
Translation of queries. Our approach is heavily reliant on obtaining a good translation. The results from Google were adequate but there were many errors which affected subsequent processing. A better strategy has to be worked out, perhaps involving a more detailed grammatical analysis of the query based on a specialised knowledge of query forms (rather than texts in general) followed by submission of individual query constituents for translation. In addition, several engines could be used and the results combined to overcome repeated errors of specific types committed by particular systems.
Translation of names within queries. Some names were recognised by Google (e.g. ‘États-Unis’ ) and hence translated correctly (e.g. ‘United States’ ). However, many were left untranslated (e.g. Tchétchénie). The document collection refers to names in their American English form which can be radically different from the French term. For example the equivalent of Tchétchénie is Chechnya though it can appear in other variants. Generating the correct translation(s) for the whole range of names which might apear in a factoid question is a major task which we have not yet looked at in detail. Without accurate translations we will not answer questions mentioning such names correctly.
Diacriticals. Our approach to these was to leave them in place for translation but to remove them entirely before question classification. This unsophisticated approach must surely lose important information. The processing of texts using diacriticals raises issues which we have not previously encountered when working with English documents. The most obvious of these is the need to match a word containing the appropriate accents with its equivalent without, while at the same time giving priority to exact matches. The present cross-language system was essentially a monolingual English one with a French front end but in the context of monolingual French-French QA attention would have to be paid to query formulation unless the retrieval engine used explicitly supported a morphologically sophisticated model of multilingual documents. Search queries returning no results. Intentionally narrow search expressions submitted to DTSearch could result in no matched document portions. This happened more often than we anticipated, the main reason being incorrectly translated (or not translated) parts of the input query. The answer is better translation together with query relaxation.
General names. Our experiment in recognising general names was not very successful. Part of the problem is accurate verification of candidates without knowing the question type. A better strategy has to be worked out for this as there will always be unknown questions.
Clarke, C. L. A., Cormack, G. V., Kemkes, G., Laszlo, M., Lynam, T. R., Terra, E. L., & Tilker P.L. (2003). Statistical Selection of Exact Answers (MultiText Experiments for TREC 2002). In E. M. Voorhees and L. P. Buckland (Eds) Proceedings of the Eleventh Text REtrieval Conference (TREC 2002), Gaithersburg, Maryland, November 19-22, 2002. NIST Special Publication 500-251. Gaithersburg, MD: Department of Commerce, National Institute of Standards and Technology.
DTSearch (2000). www.dtsearch.com .
Google Translation (2003). www.google.ie/language_tools?hl=en . O’Gorman, A. (2003). Open Domain Question Answering in French. Undergraduate Final Year Project, University of Limerick, Ireland.