European Web Retrieval Experiments at WebCLEF 2006 Stephen Tomlinson stephen.tomlinson@hummingbird.com 0 Ottawa , Ontario , Canada

Hummingbird participated in the WebCLEF mixed monolingual retrieval task of the Cross-Language Evaluation Forum (CLEF) 2006. In this task, the system was given 1939 known-item queries, and the goal was to find the desired page in the 82GB EuroGOV collection (3.4 million pages crawled from government sites of 27 European domains). The 1939 queries included 124 new manually-created queries, 195 manuallycreated queries from last year, and 1620 automatically-generated queries. In our experiments, the results on the automatically-generated queries were not always predictive of the results on the manually-created queries; in particular, our title-weighting and duplicate-filtering techniques were fairly effective on the manually-created queries but were detrimental on the automatically-generated queries.

 First Relevant Score Automatically-Generated Queries Hummingbird Ottawa, Ontario, Canada stephen.tomlinson@hummingbird.com http://www.hummingbird.com/

Hummingbird SearchServer1 is a toolkit for developing enterprise search and retrieval applications. The SearchServer kernel is also embedded in other Hummingbird products for the enterprise.

SearchServer works in Unicode internally [3] and supports most of the world’s major character sets and languages. The major conferences in text retrieval experimentation (CLEF [2], NTCIR [4] and TREC [ 10 ]) have provided judged test collections for objective experimentation with SearchServer in more than a dozen languages.

This (draft) paper describes experimental work with SearchServer for the task of finding named pages in various European languages using the WebCLEF 2006 test collection.

1SearchServerTM, SearchSQLTMand Intuitive SearchingTM are trademarks of Hummingbird Ltd. All other copyrights, trademarks and tradenames are the property of their respective owners.

Methodology

For the submitted runs in July 2006, SearchServer experimental development build 7.0.1.271 was used. 2.1

Data The collection to be searched was the EuroGOV collection [ 8 ]. It consisted of 3,589,502 pages crawled from government sites of 27 European domains. Uncompressed, it was 88,062,007,676 bytes (82.0 GB). The average document size was 24,533 bytes. Note that we only indexed 3,417,463 of the pages because the organizers provided a “blacklist” of 172,039 pages to omit (primarily binary documents).

For the mixed monolingual task, there were 1939 queries, including 124 new manually-created queries, 195 manually-created queries from last year, and 1620 automatically-generated queries.

Based on the official query labels, here is the count of the number of queries of each language: – DA 15, DE 30, EN 30, ES 30, HU 15, NL 30, PT 30, RU 15. – CS 1, DA 23, DE 61, EL 14, EN 94, ES 13, ET 24, FI 27, FR 51, GA 2, HU 28, IT 12,

LT 5, LV 21, NL 14, PL 22, PT 22, RU 3, SK 13, SV 19, UNKNOWN 341. ² “manual new” topics (124 total):

– DE 30, EN 30, ES 24, HU 10, NL 30. ² “manual old” topics (195 total): ² “auto uni” topics (810 total): ² “auto bi” topics (810 total): – DA 19, DE 61, EL 17, EN 101, ES 19, ET 16, FI 21, FR 60, GA 2, HU 30, IS 1, IT 15,

LT 11, LV 24, NL 16, PL 19, PT 26, RU 3, SK 13, SV 21, UNKNOWN 315.

More details on the mixed monolingual task are presumably in the track overview paper.

 3 Indexing

Our indexing approach was similar to what we used last year (described in detail in [ 12 ]). Briefly, in addition to full-text indexing, the custom text reader cTREC populated particular columns such as TITLE (if any), URL, URL_TYPE and URL_DEPTH. The URL_TYPE was set to ROOT, SUBROOT, PATH or FILE, based on the convention which worked well in TREC 2001 for the Twente/TNO group [ 16 ] on the entry page finding task (also known as the home page finding task). The URL_DEPTH was set to a term indicating the depth of the page in the site. Table 1 contains URL types and depths for example URLs. The exact rules we used are given in [ 13 ].

We used the first recognized ‘charset’ specification in the page (e.g. from the meta http-equiv tag) to indicate from which character set to convert the page to Unicode (Win_1252 was assumed if no charset was specified).

One change from last year was the use of a new stopword list which concatenated stopword lists of 15 European languages (DA, DE, EL, EN, ES, FI, FR, HU, IT, NL, NO, PT, RU, SV, TR).

The apostrophe was treated as a term separator. No accents were indexed. Stemming was not used for any of our runs this year. We executed 6 runs in July 2006, though only 5 were allowed to be submitted. All 6 are described here.

humWC06nos: This run was the same as humWC06 (described below) except that no stopword list was used. (This run was not submitted.)

humWC06: This submitted run was a plain content search of the baseline table. It used the ‘2:3’ relevance method and document length normalization (SET RELEVANCE_DLEN_IMP 500). Below is an example SearchSQL query:

SELECT RELEVANCE(’2:3’) AS REL, DOCNO FROM EGOV WHERE

(FT_TEXT IS_ABOUT ’Giuseppe Medici’) ORDER BY REL DESC; humWC06p run: This submitted run was the same as humWC06 except that it put additional weight on matches in the title, url, first heading and some meta tags, including extra weight on matching the query as a phrase in these fields. Below is an example SearchSQL query. The searches on the ALL_PROPS column (which contained a copy of the title, url, etc. as described in [ 13 ]) are the difference from the humWC06 run. Note that the FT_TEXT column indexed the content and also all of the non-content fields except for the URL. Unlike last year, we used WEIGHT 2 instead of WEIGHT 1 for the “ALL_PROPS IS_ABOUT” weight: SELECT RELEVANCE(’2:3’) AS REL, DOCNO FROM EGOV WHERE (ALL_PROPS CONTAINS ’Giuseppe Medici’ WEIGHT 1) OR (ALL_PROPS IS_ABOUT ’Giuseppe Medici’ WEIGHT 2) OR (FT_TEXT IS_ABOUT ’Giuseppe Medici’ WEIGHT 10) ORDER BY REL DESC; humWC06dp run: This submitted run was the same as humWC06p except that it put additional weight on urls of depth 4 or less. Less deep urls also received higher weight from inverse document frequency because (presumably) they were less common. Below is an example WHERE clause: WHERE ((ALL_PROPS CONTAINS ’Giuseppe Medici’ WEIGHT 1) OR (ALL_PROPS IS_ABOUT ’Giuseppe Medici’ WEIGHT 2) OR (FT_TEXT IS_ABOUT ’Giuseppe Medici’ WEIGHT 10) ) AND ( (URL_TYPE CONTAINS ’ROOT’ WEIGHT 0) OR (URL_TYPE CONTAINS ’SUBROOT’ WEIGHT 0) OR (URL_TYPE CONTAINS ’PATH’ WEIGHT 0) OR (URL_TYPE CONTAINS ’FILE’ WEIGHT 0) OR (URL_DEPTH CONTAINS ’URLDEPTHA’ WEIGHT 5) OR (URL_DEPTH CONTAINS ’URLDEPTHAB’ WEIGHT 5) OR (URL_DEPTH CONTAINS ’URLDEPTHABC’ WEIGHT 5) OR (URL_DEPTH CONTAINS ’URLDEPTHABCD’ WEIGHT 5) ) humWC06dpc run: This submitted run was the same as humWC06dp except that it applied an experimental duplicate-filtering heuristic.

humWC06dpcD run: This run was the same as humWC06dpc except that the domain information of the topic metadata was used to restrict the search to the specified domain. Below is an example of the domain filter added to the WHERE clause for a case in which the page was known to be in the ‘it’ domain (which implied the DOCNO would contain ‘Eit’).

AND (DOCNO CONTAINS ’Eit’ WEIGHT 0) 4

Results of Web Search Experiments The 6 runs allow us to isolate 5 ‘web techniques’ which are denoted as follows: ² ‘s’ (stopwords): The humWC06 score minus the humWC06nos score. ² ‘p’ (extra weight for phrases in the Title and other properties plus extra weight for vector search on properties): The humWC06p score minus the humWC06 score. ² ‘d’ (modest extra weight for less deep urls): The humWC06dp score minus the humWC06p score. ² ‘c’ (duplicate-filtering): The humWC06dpc score minus the humWC06dp score. ² ‘D’ (domain filtering): The humWC06dpcD score minus the humWC06dpc score. ² “new”: the 124 new manually-created topics ² “old”: the 195 manually-created topics from last year ² “uni”: the 810 automatically-generated “auto uni” topics ² “bi”: the 810 automatically-generated “auto bi” topics.

Table 3 isolates the differences in Generalized Success@10 (GS10) between the runs of Table 2. (Details of the column headings can be found in our companion ad hoc paper [ 11 ].) For a topic, GS10 is 1:081¡r where r is the rank of the first row for which a desired page is found, or zero if a desired page was not found. Last year [ 12 ], GS10 was known as “First Relevant Score” (FRS).

Preliminary findings from Table 3 include the following: ² The ‘s’ technique (stopwords) was not as beneficial on the new topics as last year’s topics.

We have not yet had time to investigate particular topics to find out why not. ² The ‘p’ technique (extra weight for phrases in the Title and other properties plus extra weight for vector search on properties), which has been reliably effective on the manually-created queries over the years, was detrimental on the automatically-generated queries. Run humWC06dpcD humWC06dpc humWC06dp humWC06p humWC06 (humWC06nos) old topics: humWC06dpcD humWC06dpc humWC06dp humWC06p humWC06 (humWC06nos) “auto uni” topics: humWC06dpcD humWC06dpc humWC06dp humWC06p humWC06 (humWC06nos) “auto bi” topics: humWC06dpcD humWC06dpc humWC06dp humWC06p humWC06 (humWC06nos) 0.510 0.494 0.497 0.499 0.466 0.463 0.463 0.455 0.447 0.425 0.390 0.377 0.072 0.048 0.056 0.056 0.057 0.057 0.685 0.657 0.665 0.666 0.648 0.655 0.622 0.610 0.600 0.571 0.528 0.524 0.123 0.089 0.115 0.113 0.116 0.115 [1] AltaVista’s Babel Fish Translation Service. http://babelfish.altavista.com/tr [2] Cross-Language Evaluation Forum web site. http://www.clef-campaign.org/ [3] Andrew Hodgson. Converting the Fulcrum Search Engine to Unicode. Sixteenth International

Unicode Conference, 2000. [4] NTCIR (NII-NACSIS Test Collection http://research.nii.ac.jp/»ntcadm/index-en.html for

IR

 Systems) Home Page. October 2001. [6] S. E. Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu and M. Gatford. Okapi at

Multilingual information retrieval resource with Hummingbird

 [5] M. F. Porter . Snowball: A language for stemming http://snowball .tartarus.org/texts/introduction.html [6] S. E. Robertson , S. Walker , S. Jones , M. M. Hancock-Beaulieu and M. Gatford . Okapi at TREC-3. Proceedings of TREC-3 , 1995 . [7] Jacques Savoy . CLEF and http://www.unine.ch/info/clef/ [8] Börkur Sigurbjörnsson , Jaap Kamps and Maarten de Rijke. EuroGOV: Engineering a Multilingual Web Corpus . Working Notes of CLEF 2005 . [9] Börkur Sigurbjörnsson , Jaap Kamps and Maarten de Rijke. Overview of WebCLEF 2005. Working Notes of CLEF 2005 . [10] Text REtrieval Conference (TREC) Home Page . http://trec.nist.gov/ [11] Stephen Tomlinson . Comparing the Robustness of Expansion Techniques and Retrieval Measures . To appear in Working Notes of CLEF 2006 . [12] Stephen Tomlinson . European Web Retrieval Experiments SearchServerTM at CLEF 2005. Working Notes of CLEF 2005 . [13] Stephen Tomlinson . Experiments in Named Page Finding and Arabic Retrieval with Hummingbird SearchServerTM at TREC 2002 . Proceedings of TREC 2002 . [14] Stephen Tomlinson . Robust, Web and Genomic Retrieval with Hummingbird SearchServerTM at TREC 2003 . Proceedings of TREC 2003 . [15] Stephen Tomlinson . Robust, Web and Terabyte Retrieval with Hummingbird SearchServerTM at TREC 2004 . Proceedings of TREC 2004 . [16] Thijs Westerveld , Wessel Kraaij and Djoerd Hiemstra . Retrieving Web Pages using Content, Links, URLs and Anchors . Proceedings of TREC 2001 .