Our first objective in participating in this domain-specific evaluation campaign is to propose and evaluate various indexing and search strategies for the German, English and Russian languages, in an effort to obtain better retrieval effectiveness than that of the language-independent approach (n-gram). To do so we evaluate the GIRT-4 test-collection using the Okapi, various IR models derived from the Divergence from Randomness (DFR) paradigm, the statistical language model (LM) together with the classical tf.idf vector-processing scheme.
Domain-specific retrieval is an interesting task, one in which we access bibliographic notices (usually composed of a title and an abstract) extracted from two German social science sources and one Russian corpus. The records in these notices also contain manually assigned keywords extracted from a controlled vocabulary by librarians who are knowledgeable of the discipline to which the indexed articles belong. These descriptors should be helpful in improving document surrogates and consequently the extraction of more pertinent information, while also discarding irrelevant abstracts. Access to the underlying thesaurus would also improve retrieval performance.
The rest of this paper is organized as follows: Section 2 describes the main characteristics of the GIRT-4 (written in the German and English languages) and ISISS (Russian) test-collections. Section 3 outlines the main aspects of our stopword lists and light stemming procedures, along with the IR models used in our experiments. Section 4 explains different blind query expansion approaches and evaluates their use with the available corpora. Section 5 provides our official runs and results.
In the domain-specific retrieval task, the two available corpora are composed of bibliographic records extracted
from various sources in the social sciences domain. Typical records (see Figure 1 for a German example) in this
corpus consist of a title (tag <TITLE-DE>), author name (tag <AUTHOR>), document language (tag
<LANGUAGE-CODE>), publication date (tag <PUBLICATION-YEAR>) and abstract (tag <ABSTRACT-DE>).
Manually assigned descriptors and classifiers are provided for all documents. An inspection of this German corpus
reveals that all bibliographic notices consist of a title and 96.4% of them include an abstract. In addition to this
information provided by the author, a typical record contains on average 10.15 descriptors
(“<CONTROLLED-TERM-DE>”), 2.02 classification terms (“<CLASSIFICATION-TEXT-DE>”), and 2.42
methodological terms (“<METHOD-TEXT-DE>“ or “<METHOD-TERM-DE>“). The manually assigned descriptors
are extracted from the controlled list known as the “Thesaurus for the Social Sciences”. Finally, associated with
each record is a unique identifier (“<DOCNO>”).
<DOC> <DOCNO> GIRT-DE19909343 <TITLE-DE> Die sozioökonomische Transformation einer Region : Das Bergische Land von 1930 bis 1960 <AUTHOR> Henne, Franz J. <AUTHOR> Geyer, Michael <PUBLICATION-YEAR> 1990 <LANGUAGE-CODE> DE <CONTROLLED-TERM-DE> Rheinland <CONTROLLED-TERM-DE> historische Entwicklung <CONTROLLED-TERM-DE> regionale Entwicklung <CONTROLLED-TERM-DE> sozioökonomische Faktoren <METHOD-TERM-DE> historisch <METHOD-TERM-DE> Aktenanalyse <CLASSIFICATION-TEXT-DE> Sozialgeschichte <ABSTRACT-DE> Die Arbeit hat das Ziel, anhand einer regionalen Studie die Entstehung des "modernen" fordistischen Wirtschaftssystems und des sozialen Systems im Zeitraum zwischen 1930 und 1960 zu beleuchten; dabei geht es auch um das Studium des "Sozial-imaginären", der Veränderung von Bewußtsein und Selbst-Verständnis von Arbeitern durch das Erlebnis und die Erfahrung der Depression, des Nationalsozialismus und der Nachkriegszeit, welches sich in den 1950er Jahren gemeinsam mit der wirtschaftlichen Veränderung zu einem neuen "System" zusammenfügt. <DOC> …
The above-mentioned German collection was translated into British English, mainly by professional translators whose native language was English. Included in all English records is a translated title (listed under “<TITLE-EN>” in Figure 2), manually assigned descriptors (“<CONTROLLED-TERM-EN>”), classification terms (“<CLASSIFICATION-TEXT-EN>”) and methodological terms (“<METHOD-TERM-EN>”). Abstracts however were not always translated (in fact they are available for only around 15% of the English records).
In addition to this bilingual corpus, we may also access the GIRT thesaurus, containing 10,623 entries (all including both the <GERMAN> and <GERMAN-CAPS>) tags together with 9,705 English translations. It also contains 2,947 <BROADER-TERM> relationships and 2,853 <NARROWER-TERM> links. The synonym relationship between terms is expressed through <USE-INSTEAD> (2,153) links, <RELATED-TERM> (1,528) or <USE-COMBINATION> (3,263).
As a third language, we access bibliographic records written in the Russian language composed of the ISISS (Russian Economic and Social Science) bibliographic data collection (see Figure 3 for an example of a record extracted from the Russian collection). Using a pattern similar to that of the other two corpora, records include a title (“<TITLE-RU>” in Figure 3), sometimes an abstract (“<ABSTRACT-RU>”), and certain manually assigned descriptors (“<KEYWORDS-RU>”).
During the indexing process, we retained all pertinent sections in order to build document representatives. Additional information such as author name, publication date and the language in which the bibliographic notice was written are of less importance, particularly from an IR perspective, and thus they will be ignored in our experiments.
As shown in Appendix 2, the available topics cover various subjects (e.g., Topic #206: “Environmental justice,” Topic #209: “Doping and sports,” Topic #221: “Violence in schools,” or Topic #211: “Shrinking cities”), and some of them may cover a relative large domain (e.g. Topic #212: “Labor market and migration”).
For the English, German and Russian language, we used the same stopword lists and stemmers that we selected
for our previous CLEF participation
In order to obtain a broader view of the relative merit of various retrieval models, we may first adopt the classical tf idf indexing scheme. In this case, the weight attached to each indexing term in a document surrogate (or in a query) combines the term's occurrence frequency (denoted tfij for indexing term tj in document Di) and also the inverse document frequency (denoted idfj).
In addition to this vector-processing model, we may also consider probabilistic models such as the Okapi model
(or BM25)
wij = Inf1ij · Inf2ij = –log2[Prob1 ij(tf)] · (1 – Prob2ij(tf))
Prob1ij = (e-λj · λtfij) / tfij!
with λj = tcj / n Prob2ij = 1- [(tcj+1) / (df j · (tfnij+1))]
with tfnij = tfij · log2[1 + ((c · mean dl) / li) where tcj represents the number of occurrences of term tj in the collection, dfj the number of documents in which the term tj appears, and n the number of documents in the corpus. Moreover, c and mean dl (average document length) are constants whose values are given in the Appendix 1.
Prob1ij = [1 / (1+λj)] · [λj / (1+λj)]tfnij Prob2ij = tfnij / (tfnij + 1)
Inf1ij = tfnij · log2[(n+1) / (dfj+0.5)]
For the third model I(n)B2, we still use Equation 2 to compute Prob2ij but the implementation of Inf1ij is modified as:
For the fourth model I(ne)C2 the initial value of Prob2ij is obtained from Equation 2 and for the value Inf1ij we use:
Inf1ij = tfnij · log2[(n+1) / (ne+0.5)]
with ne = n · [1 - [(n-1) / n]tcj]
Finally, we also consider an approach based on a statistical language model (LM)
P[Di | Q] = P[Di] . ∏tj∈Q [λj . P[tj | Di] + (1-λj) . P[tj | C]] with P[tj | Di] = tfij/li and P[tj | C] = dfj/lc with lc = ∑k dfk where λj is a smoothing factor (constant for all indexing terms tj, and usually fixed at 0.35) and lc an estimate of the size of the corpus C.
To measure the retrieval performance, we adopted the mean average precision (MAP) (computed on the basis of 1,000 retrieved items per request by the new TREC-EVAL program). In the following tables, the best performances under the given conditions (with the same indexing scheme and the same collection) are listed in bold type.
From this table, we can see that when using word-based indexing, the DFR I(ne)B2 or the LM models tend to perform the best. With the 4-gram indexing approach, the LM model always presents the best performing schemes. The short query formulation (T) tends to produce a better retrieval performance than medium (TD) topic formulation. As shown in the last line, when comparing the word-based and 4-gram indexing systems, the relative difference is seen to be rather short (around 4.6%) and favors the 4-gram approach.
Using our evaluation approach, evaluation differences occur when comparing with values computed according to the official measure (the latter always takes 25 queries into account).
Indexing / stemmer IR Model DFR GL2 DFR I(ne)B2 LM (λ=0.35) Okapi tf idf Mean % change over T over stemming
To provide a better match between user information needs and documents, various query expansion techniques
have been suggested. The general principle is to expand the query using words or phrases having similar meanings
to, or related to those appearing in the original request. To achieve this, query expansion approaches consider
various relationships between these words, along with term selection mechanisms and term weighting schemes.
Specific answers regarding the best technique may vary, thus leading to a variety of query expansion approaches
In our first attempt to find related search terms, we might ask the user to select additional terms to be included in
an expanded query. This could be handled interactively through displaying a ranked list of retrieved items returned
by the first query. As a second strategy,
As a third technique,
In the current context, we used Rocchio’s formulation (denoted “Rocchio”) with α = 0.75, β = 0.75, whereby the system was allowed to add m terms extracted from the k best ranked documents from the original query. For the German corpus (Table 4, third column), such a search technique does not seem to enhance the MAP. For the English collection (Table 5, second and third column), Rocchio’s blind query expansion may improve the MAP from +9.3% (DFR PB2, 0.3101 vs. 0.3392) or hurt the retrieval performance -8.72% (Okapi model, 0.3039 vs. 0.2774). For the Russian language (Table 6, second and forth column), blind query expansion improves the MAP (e.g., +28.98% with the Okapi model, 0.1740 vs. 0.1349 or +2.3% with the DFR I(ne)B2 model, 0.1503 vs. 0.1468).
German German
Rocchio idf DFR I(n)B2 0.4179 DFR I(n)B2 0.4179 5/70 0.3965 5/70 0.4120 10/100 0.3965 10/100 0.4025 10/200 0.3992 10/200 0.4104
English English
Rocchio idf DFR PB2 0.3101 DFR PB2 0.3101 Rocchio's query expansion approach however does not always significantly improve the MAP. Such a query expansion approach is based on term co-occurrence data and tends to include additional terms that occur very frequently in the documents. In such cases, these additional search terms will not always be effective in discriminating between relevant and non-relevant documents, and the final effect on retrieval performance could be negative.
As another pseudo-relevance feedback technique we may apply an idf-based approach (denoted “idf” in
following tables)
For example, with the Russian language (Table 6, third and fifth column), this idf-based blind query expansion may also improve the MAP (e.g., +19.5% with the Okapi model, 0.1612) but, on the other hand, with the DFR I(ne)B2 model, the MAP is slightly reduced (-2.3% from 0.1468 to 0.1433).
However, the idf-based query expansion tends to include rare terms, without considering the context. Thus among the top-k retrieved documents such a scheme may add terms appearing far away from where the search terms occurred. The single selection criterion is based only on idf values, not the position of those additional terms in the top-ranked documents. This year we investigated retrieval effectiveness when including a second criterion in the selection of terms to be included in the new expanded query. We considered it to be important to expand the query using terms appearing close to a search term (fixed at 10 indexing terms in the current experiments). This short window includes 10 terms to the right and 10 terms to the left of each query term. This type of query expansion method is denoted as “idf-window” in Table 7.
Finally, to find words or expressions related to the current request, we considered using commercial search engines (e.g., Google) or online encyclopedia (e.g., Wikipedia). In this case, we submitted a query containing the short topic formulation (T or title-only) to each information service. When using Google, we fetched the first two text snippets and added them as additional terms to the original topic formulation, forming a new expanded query. When using Wikipedia, we fetched the first returned article and added the ten most frequent terms (tf) contained in the extracted article. The retrieval effectiveness of our two new query expansion approaches is depicted in Table 7 (German collection) and is compared to two other query expansion techniques. Compared to the performance before query expansion (0.4096), Rocchio's and the idf-based blind query expansion cannot improve the MAP. On the other hand, the variant “idf-window” presents a better retrieval performance (+4.9%, from 0.4069 to 0.4247). Using the first two text snippets returned by Google, we may also enhance slightly the MAP (from 0.4096 to 0.4196, or +2.4%). The MAP variation varied according to approaches and parameter settings, while the largest enhancement could be found using the idf+window technique (forth column in Table 7). Finally, using Google to find related terms or phrases implied that we required more processing time.
As a complementary search technique, we used two stemmers when defining the official run UniNEDSde3. In this case we first applied our light stemming approach and then a more aggressive one. If the same term was produced by the two stemmers, we only kept one occurrence. On the other hand, if the returned stem differed, we added the two forms to the query formulation. Run name Language Query UniNEDSde1 German TD
TD
TD UniNEDSde2 German TD
TD
TD UniNEDSde3 German T special TD
TD UniNEDSde4 German TD
TD
TD UniNEDSen1 English TD
TD
TD UniNEru1 Russian TD
TD
0.3770
0.1594 (0.1531) Z-score 0.1628 (0.1563)
0.1655 (0.1589) Z-score 0.1890 (0.1815)
This year we suggest two new query expansion techniques. The first, denoted "idf-window", is based on co-occurrence of relatively rare terms in a close context (within 10 terms from the occurrence of a search term in a retrieved document). As a second approach, we add the first two text snippets found by Google to expand the query. Compared to the performance before query expansion (e.g., with Okapi the MAP is 0.4096), Rocchio's and the idf-based blind query expansion cannot improve this retrieval performance. On the other hand, the variant “idf-window” presents a better retrieval performance (+4.9%, from 0.4069 to 0.4247). Using the first two text snippets returned by Google, we may also enhance slightly the MAP (from 0.4096 to 0.4196, or +2.4%).
Acknowledgments
The authors would like to also thank the GIRT - CLEF-2008 task organizers for their efforts in developing domain-specific test-collections. This research was supported in part by the Swiss National Science Foundation under Grant #200021-113273.
German GIRT English GIRT Russian word Russian 4-gram b C201 C202 C203 C204 C205 C206 C207 C208 C209 C210 C211 C212
Violence in old age Tobacco advertising Islamist parallel societies in Western Europe Poverty and social exclusion Generational differences on the Internet (Intellectually) Gifted
Employment service Chronic illnesses