EuroGOV is a multilingual web corpus that was created to serve as the document collection for WebCLEF, the CLEF 2005 web retrieval task. EuroGOV is a collection of web pages crawled from the European Union portal, European Union member state governmental web sites, and Russian government web sites. The corpus contains over 3 million documents written in more than 20 different European languages. In this paper we provide a detailed description of the EuroGOV collection.
where collection building is less restricted by intellectual property rights. The resulting collection,
which we think of as a European counterpart of the .GOV collection [
In this paper we describe the EuroGOV collection in detail. The paper is organized as follows. We start by describing the crawling process in Section 2. Section 3 then lists various characteristics of the resulting collection, including the domains, the languages, and the link structure. We conclude with some discussion and future outlook in Section 4. 2
Our initial plan for building EuroGOV was to obtain a focused crawl from the European Union seed .eu.int, and branch into the individual member states’ governmental sites. However, restricting a crawler to government-related sites proved highly non-trivial. There is no simple way to tell a European government site apart from any other European site. For some government sites the crawling is smooth and we can easily lfiter out governmental pages (notable examples include .gov.uk and .regeringen.se). Most governmental sites, however, have more complex structures, and we could only focus the crawl by providing an explicit list of domains. As an example, we initially crawled 13 different domains to gather pages from the Finnish government. As the following domain list shows, there is no easy way of identifying Finnish governmental domains: defmin.fi, formin.finland.fi, intermin.fi, ktm.fi, minedu.fi, mintc.fi, mmm.fi, mol.fi, om.fi, stm.fi, vm.fi, vnk.fi, and ymparisto.fi These differences in domain naming traditions will make it difficult to guarantee completeness of some governments. As a result, what we should realistically aim for is that EuroGOV contains the fairly complete content of • the main government portals, and • the main ministries of the countries on whose information we want to include in the corpus.
Our crawling process can be divided into three parts. Our initial seed was made by picking 2–3 main governmental sites for each EU member state. The seed contained 40 URLs and was created by referring to a list from the EU portal.1 After completing several cycles of this crawl we realized that due to the varying structure of governmental sites, the portion of governmental pages covered differed considerably from one country to another. In order to try to get a better harmony in coverage we began a new crawl, now starting with a seed consisting of a list of ministries for a subset of the EU countries. The subset covered 12 countries and was chosen according to CLEF interests. The left column of Table 1 shows the list of main domains. The second seed list consisted of 131 ministries from 9 EU member states (UK, Sweden, and EU itself were considered adequately covered in the initial crawl). The seed was created by browsing the main government portals. The third crawl was performed when interest was expressed in including Russian government pages in the crawl. The Russian crawl was created from a single seed: www.gov.ru. The final collection was created by combining the three crawls into a single collection of pages. 3
In this section we provide various statistics concerning the collection, including the domains covered, the languages it contains, and its link structure. 3.1
The EuroGOV collection has pages from the 27 primary domains listed in Table 1. There is a set of 13 main domains, shown on the left-hand side of Table 1, chosen in accordance with current CLEF interests and plans. We have attempted to include a sufficiently large number of pages from these 13 main domain. There are 14 additional domains, shown on the right-hand side of Table 1, from which pages are also included in the collection. The coverage of these additional domains is often less complete than the coverage of the main domains. Note that pages in the languages of the additional domains will ‘creep in’ anyway. For example, the eu.int domain has ample pages in all of the 20 official languages of the European Union.
The EuroGOV collection features more languages and countries than are being used in the WebCLEF 2005 evaluation tasks. We made a deliberate choice to go for this extended list of countries and domains. This will facilitate future task extensions for cross-lingual web retrieval, or re-use of the collection for other purposes. We also feel that this reeflcts the natural situation when building a ‘European’ search engine.
The EuroGOV collection contains a total of 3,589,501 pages, and can be compressed in 11 gigabytes of data. Table 2 gives the page counts for each of the primary domains in the collection. The rfist column lists the primary domains in the collection. The second through fifth columns list the total number of web pages per domain; the number of MD5 checksums (of the page’s content) that occur more than once; the number of pages that have a repeated MD5 checksum (and thus the same content as another page); and the number of unique pages. The final, sixth, column lists the total size of the pages when compressed. The vfie domains with the largest numbers of pages are: Finland (661,599), Germany (444,794), European Union (374,484), Hungary (330,822), Czech Republic (324,496). Although the number of pages per domain varies between 661,559 (Finland) and 303 (Greece), the number of pages is generally sufficient to support the building of a test collection. Specifically, the smallest set of pages for one of the main domains is 35,168 (Spain). It is unclear, at this point, to what extent the varying numbers of pages per domain is a result of the available web content, different link structure of different governmental sites, or of our particular choices in crawler software or seed points.
What is the distribution of languages in the collection? To answer this question, we applied the
TextCat language identicfiation tool [
When looking at the distribution of languages over the whole collection, shown on the lefthand side of Table 3, we see that the most frequent languages are Finnish (20%), German (18%), Hungarian (13%), English (10%), and Latvian (9%). It is a surprizing outcome that languages of the Finno-Ugrian family dominate the collection! The distribution of languages over the collection closely corresponds with the number of pages per domain (in numbers of pages, Finnish ranked first and Hungarian ranked fourth, see Table 2). A look at the distribution of languages for Germany, France, and the UK, shown in the center of Table 3, confirms this strong correlation between country and official language: In the German domain, 98% of the pages is in German. In the French domain, 94% of the pages are in French, and in the UK domain, 99% of the pages are 1URL: http://europa.eu.int/abc/governments/index_en.htm in English. In countries with more than one official language, such as Finland (with Finnish and Swedish) or Belgium (with Dutch, French, and German), we see more language diversity within the corresponding domains. The language distribution for the Finnish and Belgian domains is also shown in the center of Table 3. Since the languages and domains seem to be closely tied together, the distribution of the mixed language domain of the European Union, shown on the right-hand side of Table 3, is of great interest. Here, we see that English is the most used language, accounting for 33% of the pages, followed by French (18%) and German (9%). 3.3
Table 4 lists a number of salient features of the EuroGOV link structure. The second and third columns give the counts of the number of links and the number of realized links (ones whose targets are in EuroGOV; columns 4 and 5 provide the average number of links per page and the average number of realized links per page. The last row but one provides averages over all top level domains, and the last row provides the total number of links in the collection.
The largest numbers of links can be found in the domains with the largest numbers of pages; just under half of the links are realized in the collection. The average number of links per page varies considerably between domains, and the average realized number of links is just over half of the average number of links, although the relative gap between the two numbers varies quite a lot between domains, (e.g., for it the average realized fan-out is 84% of all links per page, while it is only 16% for ru). EuroGOV was thought of as an experimental collection for evaluating cross-lingual web retrieval. As such, the collection serves its purpose well. However, EuroGOV has several limitations which should be taken into account when working with the current collection and planning for possible future extensions of the collection.
• Completeness: Quite some effort was put into collecting lists of governmental sites to crawl.
This is, however, not a complete list. • Incomplete description of the link structure: A full link analysis of the EuroGOV collection has not been performed yet. This is not an inherent limitation of the document collection. However, this sort of analysis is important for evaluating whether the collection is a good representative of a realistic web. • Empty pages: The collection contains over 70,000 empty documents. It is not clear why this error ocurred, but it should be avoided in future versions of the collection.
Rich document types: In the EuroGOV collection, document types such as PDF and DOC files appear in the collection in the same format as they were crawled, i.e., their text is not extracted. Furthermore, large documents are truncated to avoid the collection growing too big. A truncated PDF or DOC lfie does not go down well with several off-the-shelf document parsers. From a web document collection perspective, this is a realistic and interesting scenario. From the perspective of a cross-lingual retrieval collection this scenario might, however, be less desirable since participants might spend too much time on these issues rather than focusing on the multi-lingual aspects of the task.
Despite its limitations EuroGOV is very suitable for the initial exploration of cross-lingual web search. 4.2
As of this moment there are no concrete plans to build an updated version of the collection. There are many interesting open questions in cross-lingual web retrieval evaluation that can be formulated and answered using the current EuroGOV collection. Off the record, the aim is to have a fresh collection ready for WebCLEF 2007 or 2008, assuming that there is interest to continue the task. As well as being a fresh crawl, the new collection should address the limitations listed above: • Improved completeness: The collection builders should try to compile a more complete list of the governmental sites of the European countries. This should ideally be performed in cooperation with uflent speakers of the languages involved.
Rich document types: Given that the main purpose of the collection is to form a document set for cross-lingual web retrieval, it should give as realistic as possible view of web data, while at the same time trying to make it relatively easy to use by cross-lingual retrieval researchers. 5
Thanks to Craig Macdonald for advise at various stages during the creation of EuroGOV, to the Melange group for sharing their normalized links with us, and to Krisztian Balog for help with exploring EuroGOV’s link structure.
The building of EuroGOV was supported by the Netherlands Organization for Scientific Research (NWO) under project numbers 017.001.190, 220-80-001, 264-70-050, 354-20-005, 612-13001, 612.000.106, 612.000.207, 612.066.302, 612.069.006, and 640.001.501.