The third participation of the University of Hagen at the German Indexing and Retrieval Test (GIRT) task of the Cross Language Evaluation Campaign (CLEF 2005) aims at providing a better baseline for experiments with natural language processing (NLP) methods in domainspecific information retrieval (IR). Our monolingual experiments with the German document collection are based on a setup combining several methods to achieve a better performance. The setup includes an entry vocabulary module (EVM), query expansion with semantically related concepts, and a blind feedback technique. The monolingual experiments focus on comparing two techniques for constructing database queries: creating a 'bag of words' and creating a semantic network by means of a syntactico-semantic parser for a deep linguistic analysis of the query. The best performance in the official experiments was achieved by a setup using staged logistic regression, a query expansion with semantically related concepts, an entry vocabulary module, a deep linguistic analysis of the query, and blind feedback (0.2875 mean average precision (MAP)). Additional experiments showed a performance improvement when changing to the basic Okapi BM25 search (0.3878 MAP). For the bilingual experiments, the English topics are translated into German queries with several machine translation services available online (Systran, Free translation, WorldLingo, and Promt). Each set of translated topics is processed separately with the same techniques as in the monolingual experiments. The best performance was achieved with a query translation by Promt with a simple keyword extraction from the translation (0.2399 MAP with a staged logistic regression approach vs. 0.2807 MAP with Okapi BM25).
1
Natural language processing for information retrieval, Deep linguistic analysis, Keyword extraction
This paper presents the work for the third participation of the University of Hagen in the domain-specific
GIRT task in the evaluation campaign of the Cross Language Evaluation Forum (CLEF). Natural language
processing (NLP) as described in the following subsections is part of query processing for the NLI-Z39.501
In CLEF 2003, retrieval strategies based on generating query variants for a single natural language (NL)
topic were compared
In CLEF 2004, the focus of our experiments was on investigating differences in indexing methods, such
as indexing unprocessed word forms, indexing concepts, and indexing semantic networks
For the monolingual experiments in CLEF 2005 two main objectives are pursued: 1. To establish a better baseline for a comparison between NLP methods and traditional approaches in IR and to achieve a better overall performance. To complete this goal, some methods from the experimental setup of the UC Berkeley in previous years were adapted. 2. To compare two techniques for creating database queries from the natural language topics: a) extracting keywords (’bag of words’) and b) applying a deep linguistic analysis by means of a syntacticosemantic parser before creating a database query.
For the bilingual experiments with the GIRT collection, the English query topics are translated into German by means of several free machine translation services that are available over the Internet. The translations obtained from each machine translation service are processed in separate experiments, employing the query processing techniques for monolingual experiments described above. 1.1
To state and investigate a hypothesis such as “The use of semantic networks (or NLP in general) – including methods such as expanding queries with semantically related concepts – improves performance in IR.” an acceptable experimental baseline has to be established.
The experimental setup used by the University of Hagen in previous participations at CLEF did not provide a performance higher than other state-of-the-art systems. It served mainly as a basis for comparing different strategies on the same system (intra-system comparison) and did not typically aim at an overall superior performance (inter-system comparison).
To obtain a better baseline for our experiments, several methods of the experiments of the UC Berkeley for CLEF were added to our experimental setup to improve retrieval performance, including a so-called entry vocabulary module. In total, our setup now includes the following options:
1The NLI-Z39.50 is being developed as part of the project “Natu¨rlichsprachliches Interface fu¨r die internationale Standardschnittstelle Z39.50” and funded by the DFG (Deutsche Forschungsgemeinschaft) within the support program for libraries “Modernisierung und Rationalisierung in wissenschaftlichen Bibliotheken”.
• LA: extracting keywords (’bag of words’) or applying a deep linguistic analysis, i.e. employing NLP
methods to produce a semantic network representation (described in
• The Cheshire II database
Two techniques for query processing are compared. The first technique corresponds to extracting keywords
from the topic title and topic description to create a database query. The query string is tokenized and word
forms are extracted. Some normalization steps such as stopword removal and stemming are employed
to produce a database query in the Database Independent Query Representation (DIQR, see
The second query processing technique employs NLP methods to create a database query. A
syntacticosemantic parser, WOCADI
For both techniques, the DIQR is mapped to a query in a formal language the database management software supports (such as a query for the Z39.50 protocol) which is then submitted to the target database.
For the GIRT task in 2005, several experimental runs for the monolingual GIRT task were submitted. The experiments vary in the following parameter settings: using a query expansion with semantically related terms (QEX=yes/no), using an entry vocabulary module (EVM=yes/no), constructing a query from the semantic network after a linguistic analysis or using a traditional keyword extraction (LA=yes/no), and using blind feedback (BF=yes/no). Table 1 gives an overview of the monolingual experiments performed for the German document collection with their results. 2.2
The best monolingual GIRT experiment used a query expansion with semantically related concepts, an entry vocabulary module, and blind feedback. The performance of the best official experiment with respect to mean average precision (0.3031 MAP for run FUHggyydbf) is better in comparison to our experiments in CLEF 2003 (0.2064 MAP) and CLEF 2004 (0.2482 MAP). Additional experiments were conducted using the state-of-the-art weighting approach, Okapi BM25, which yield a significantly better performance (0.3878 MAP for run FUHggyylbf).
The effect of any single query processing method (corresponding to a single parameter) is still inconclusive, but the combination of all processing methods with a deep linguistic analysis of the query yields the best performance with respect to the number of relevant and retrieved documents and mean average precision (FUHggyylbf).
Several differences between our setup and the experimental setup of the UC Berkeley still remain and may account for a difference in performance.
•
In the setup of the UC Berkeley, the stemmer is applied to word forms. In our setup, stemming is applied after lemmatizing. The WOCADI parser returns morpho-lexical results for the sentences, including the lemmata (complex named entities and multi-word lexemes are not identified in the morpho-lexical analysis at the moment). The input data for the German stemmer consists of these lemmata from the morpho-lexical analysis of a sentence, i.e. the stemming process starts with normalized word forms. Plural forms of nouns, participles, past forms of verbs and verb with a separable prefix are already normalized. • Database Index:
In the setup of the UC Berkeley, only the constituents of German compounds are used as index terms. In our setup, compounds are indexed together with their constituents (because of a later query expansion with semantically related compounds). In addition, our database index contains stemmed word forms from the title and the abstract fields of the documents only; a database index containing subject terms as well was not created. • Phrases:
In our setup, a simple frequency-based phrase recognition for the EVM is employed which identifies adjective-noun sequences (English noun phrases often correspond to compounds in German). • Compounds:
In the UC Berkeley setup, decompounding of compounds is based on a statistical approach
In our setup, the EVM uses a co-occurrence between title words and subject terms only and does not include words from the abstract.
These differences can account for a significantly lower performance of our experiments. For example, we conducted tests which showed a different performance for indexing lemmata and indexing stemmed word forms. Experiments on the index with stemmed word forms resulted in a significantly better performance because the stemmed forms conflate adjectives and nouns into the same form (e.g. ‘wirtschaftlich’/‘economical’ and ‘Wirtschaft’/‘economy’ are conflated into ‘wirtschaft’/‘economi’). For a database index with lemmata, only inflectional endings are removed and e.g. ‘wirtschaftlich’ and ‘wirtschaft’ remain different index terms. While treating different concepts differently remains one of our objectives for a high precision information search, our background knowledge which semantically links the correct senses of ‘Wirtschaft’ and ‘wirtschaftlich’ is neither robust (i.e., it relies on correct word sense disambiguation) nor lexically complete, i.e. the linking between corresponding concepts (derivational links) in our computational lexicon is not complete, yet. 3
In CLEF 2004, the bilingual experiments of the University of Hagen were based on a method which combines a concept translation using linguistic resources (such as GermaNet and EuroWordNet) with a word translation using translation lists to obtain a ranked list of translation alternatives. This method was employed to translate German concepts in the DIQR into English to create a database query. For CLEF 2005, English query topics are submitted to freely available machine translation services for a translation into German queries. 3.1
Our second participation in the bilingual GIRT task (matching English topics against the German data) is based on various machine translation services for a translation. For the bilingual retrieval experiments 3A ‘Fuge’ does not correspond to a morphological suffix and can not be treated by a suffix elimination process (stemming). (English – German) with the GIRT document collection, four machine translation services translate the English topics into German queries: Free translation4, Systran5, and WorldLingo6, and Promt7.
For the official runs, keywords are extracted from the translated queries: the query is tokenized, stopwords are removed, and a stemmer is applied. The remaining stemmed words are employed to create a simple DIQR, which is processed accordingly. For a fifth experiment, the topics are translated by the Promt translator, analyzed by WOCADI to produce a semantic network representation, and transformed into a DIQR with the Rete-based transformation. The DIQR queries are then processed as described in Section 2. Table 2 gives an overview of the bilingual experiments with English topics on the German document collection together with their results. 3.2
The WOCADI parser may serve as an indicator for the quality of the translated queries. WOCADI performs many morpho-syntactical and semantical tests while trying to produce a semantic network representation for a query topic. If some of these tests (such as agreement between subject and verb, selectional restrictions for the complement of an action (verb), etc.) fail or if the translation contains other errors, a partial semantic network is returned. For severe errors in the translations, no network is returned at all.
WOCADI reported no problems in parsing the original German topics. Parse results for the translated German queries showed that many of the 200 translations (25 topics, four different machine translations services, and translation of title and description) contained a syntactical or lexical error, so that WOCADI could not fully analyze these translations. For example, all services had problems translating the imperative forms of the verb ‘find’. (The exclamation mark to indicate the imperative is missing in all topics and the original queries were not modified by us.)
Due to these test results, we judge the quality of online translations so poor that the results from translating English topics into German can not be processed successfully with a deep linguistic analysis. Thus, the effect of trying to parse a grammatically incorrect translation (a partial semantic network or no semantic network is returned by WOCADI) is not present in a simple keyword extraction from a shallow analysis (the morpho-lexical stage in the WOCADI parser). All potential advantages of a deep linguistic processing are then unavailable (FUHegpyyn vs. FUHegpyyl). A performance increase was observed for all experiments using Okapi BM25.
4http://www.freetranslation.com/ 5http://www.systransoft.com/ 6http://www.worldlingo.com/ 7http://www.e-promt.com/en/
In comparison with the results for the monolingual GIRT task in 2003 and 2004, performance with respect
to the MAP for the best official experiment has improved considerably
The method for constructing a database query using the transformation of the semantic network representation into a database query yields a higher performance than extracting keywords in combination with all other methods applied in these experiments. Results are still inconclusive in which cases NLP methods provide a better performance and even seem to depend on the ranking scheme employed.
The machine translation services tested did not produce high-quality translations. At the moment, using a robust keyword extraction yields better performance than a subsequent try to semantically analyze malformed ‘translations’.
The experimental setup of the University of Hagen and of the UC Berkeley still differ much, despite using methods which are comparable, similar or even the same with respect to some experimental parameters for the monolingual experiments. Further research is required and should properly identify the differences in these parameters and their influence on the overall retrieval performance.