The objective of the experiment described in the article was the evaluation of Polish digital resources searchable by the engine Europeana. Queries were created so as to represent proportional shares of common names, named entities and their combinations. They were manually enriched by students, experienced professionals and humanities-oriented educated persons. The system responses were then evaluated according to their Mean Average Precision. The average efficiency of information retrieval for Polish monolingual queries was weak: there were only 26.6% of highly relevant responses and as much as 73.5% of queries produced unsatisfactory results. MAP produced the best results for the automatic search (0,314), while enriched files contained less relevant results: for the expert users MAP value was 0,1795, for the educated users it was 0,1529 and for students 0,1279. The overall results proved that the IR process concerning Polish resources searchable by Europeana require significant improvement.
Europeana is an ambitious project intended to integrate the Old Continent’s
digitised heritage preserved in multiple national collections and to deliver information
about its content in users’ languages [
Europeana covers resources in thirty languages. Items or objects qualified as
“Polish” are an important and constantly growing part of this offer as they constitute at
present the ninth richest subset of the entire collection. Regarding this issue from the
user’s perspective one should say that native speakers of Polish – thus potential
Europeana audience – are the fifth population of Europe, if only languages using Latin
script are considered [
The objective of the experiment described below was to evaluate the effectiveness of information retrieval conducted on Polish digital resources with the help of the engine Europeana. By “Polish” items we understand documents of any kind, prepared in any language and of any origin (not only Polish), offered by libraries, museums, documentation centres etc. located in Poland. 2
Polish is not commonly used in information retrieval systems and some
information about its lexis and morphology is necessary here. As there exists a number of
grammars of this language here only the features relevant for IR activities will be
highlighted [
Polish lexical resources have a large number of common Indo-European stems and lexemes. There are numerous words of Latin and Greek origin in the scientific terminology. A number of direct or indirect German borrowings entered Polish in the Middle Ages and in the nineteenth century. Polish borrowed also hundreds of words from Italian (sixteenth century) and from French (eighteen century). Although spelling of these words is usually adjusted to the phonology and morphology of the “target language”, their etymology is rather an advantage to users in a multilingual environment. A separate issue is spelling used in old documents and partly in their descriptions. Systems are not prepared to lemmatize or disambiguate old orthography but users who intend to work on such documents are usually familiar with the problem.
What seems to be the main challenge for the IR systems handling resources in Polish is undoubtedly morphology. Polish nouns and adjectives are inflected in number, case and gender. Polish verbs are conjugated according to person and tense. Verb tenses are additionally modified by aspect (e.g. perfect, iterative etc.) which is a systemic feature of Polish, contrary to Romance and Germanic languages. All the parts of speech have also developed a rich derivational system. Surprisingly for the users of western languages even proper names are inflected in Polish. This concerns all the nomina propria of Slavic origin and most of the borrowed ones.
There is a visible tendency to use analytical forms in Polish which means that grammatical compound morphemes are more and more often replaced by prepositions. For instance the expression send to the president could be rendered either as wysłać prezydentowi or as wysłać do prezydenta but the second form using the preposition do (more analytical) becomes prevalent. In spite of this tendency information retrieval systems based on natural language processing should consider the aforementioned characteristics of Polish as a highly flexional language. While verbs in document descriptions are often avoided or reduced to basic forms (present tense, third person, no aspect) because they do not carry significant information, nominal elements require special attention. They should be recognized and disambiguated with regard to grammatical case and number. It should be reminded that the most frequent cases in Polish are genitive, accusative and dative. Since several nouns may produce complex semantic derivatives the system should also interpret them properly. E.g. from the city name Lublin on can derive the adjective lubelski, the nouns lublinianin, lublinianka (male or female inhabitant of Lublin), and the region name lubelszczyzna, all of them inflected by number and case. But disambiguation of search terms includes also resolution of polysemy or homonymy. In this example one should distinguish between the city name and the car make of delivery vans of the same name. 3 3.1
The so called “Polish collection” is a part of CHiC 2012, and 2013 Multilingual
collection. It consists of 1,093,705 documents contained in 1,094 files and constitutes
the ninth richest collection of all the 30 languages handled by Europeana. The whole
archive has a size of 119 MB and was made available by Europeana last year at
http://ims.dei.unipd.it/data/chic/. According to CHiC 2012 evaluation [
As described in [
To make the process of indexing more effective the following set of fields was applied in the descriptions: <dc:contributor>, <dc:creator>, <dc:source>, <dc:description>, <dc:date>, <dc:language>, <dc:subject>, <dc:title>, <dc:type>, <dc:identifier>, <dc:rights>, <dcterms:alternative>, <dcterms:created>, <europeana:country>, <europeana:language>, <europeana:type>, <europeana:uri>, <europeana:year>. An example of an object from the Polish collection is presented in this mixed XML format on the Figure 1.
Fifty queries were created so as to represent proportional shares of common names, named entities (anthroponymes, toponymes, urbonymes, names of historical events etc.) and their combinations. They included in city names (e.g. in #13 Toruń), region names (e.g. in #25 Lubelszczyzna – Lublin region), persons’ names (e.g. in #11 Józef Bem), event’s names (e.g. in #8 powstania polskie – Polish uprisings), professions’ names (e.g. in #5 inżynierzy – engineers), state names (e.g. in #5, #9, #19, #30 Polska) and artifacts names (e.g. # 5 mosty – bridges). The topics were prepared on the basis of Europeana search logs and of deductions about users interests concerning Polish cultural heritage. Simple queries were composed of two to five terms (three terms on the average).
Terms were used in singular or in plural, many of them were inflected by case; they included polysemic or homonymic words. For example in the topic obraz wsi the noun obraz means picture, painting or image in a metaphoric sense; the genitive wsi comes from nom. wieś and means of a village or of a country. However, as an abbreviation of Wojskowe Służby Informacyjne WSI may also signify military information service and produce hits in the resources of the archives of the anticommunist opposition (such as KARTA). These inconsistencies were introduced deliberately in order to simulate real users behaviour who are likely to make mistakes or act by trial and error method. As it was mentioned earlier, verbs appeared rarely in topics descriptions, mostly as present or past participles.
The content of some document fields was delivered also in English or in German in addition to Polish. For example institutions such as universities appear under Polish and English names and scientific documents are provided with short abstracts in English. German appears often in the documents published before 1945 on today’s western territories of Poland (Lower and Upper Silesia, Pomerania). For example the city of Toruń has a German name Thorn, the city of Wrocław is called in German Breslau.
Each topic was identified by the tag <identifier>, while the basic query was
provided within the tag <title>. For each query an additional field <description> was
provided. The aim of this field was to give the relevance assessors an idea about
polysemic, too general, or unclear topics. As stated in [
In the Wrocław part of the experiment topics were manually enriched by three
categories of users: students of information science, experienced professionals of
information retrieval, and humanities-oriented educated persons. In this way three series of
system responses could be generated and evaluated, apart from the basic one without
enrichment [
These groups were defined in a way to simulate preferences of real users. Actually Europeana at this stage of its development is not addressed to unprepared, general public. It was thus assumed that persons interested at the cultural heritage of the continent are likely to have some educational or professional background relevant to the field covered by the search topics they use. As the evaluation process of this kind is carried out for the first time on Polish resources and their potential users, there were no prior expectations concerning information strategies of the selected groups.
It turned out that “humanities-oriented educated” users most often enriched topics with linguistic means such as synonyms of the existing terms. Expert users who have more encyclopedic knowledge preferred narrower terms related to facts described by the existing terms. E.g. the topic Polish uprisings in Kingdom of Poland elicited at this group names of commanders or battles rather than synonyms of the noun uprising (this was typical for the “educated users”). Finally students who have rather limited encyclopedic knowledge (especially about history) and little interest at cultural heritage issues applied mixed heuristic strategies. <topic lang="en"> <identifier>CHIC-2013-PL-030</identifier> <title>Poland and Europe in the 18th century</title> <enrichment>Northern war, partitions of Poland, Russia, Austria, Prussia, Sweden, France</enrichment> <description>works on contacts and relations with European countries in XVIII century</description> </topic>
For each enriched file as well as for the collection documents stop word removing
procedure was applied. The stoplist consists of 304 most frequent and semantically
insignificant entries. The list includes determinants, prepositions, conjunctions,
pronouns (with their inflected forms). The stop words removal procedure was applied for
manually enriched topics files, as well as for the collection itself. As weighting
scheme for all the runs OKAPI (BM25) probabilistic algorithm was used [
Results have been evaluated according to the following evaluation schemes: MAP, P@5, P@10, p-value, GMAP, MFRS. Finally, as for previous CHiC experiments, Mean Average Precision (MAP) values were applied. For each topic MAP value has been computed for the first 1000 retrieved documents in a ranked list. 4
The analysis of the results led to a series of conclusions, which can be grouped in four categories: quality of descriptions inconsistencies due to the variety of items and exhibits types covered by Europeana language-specific difficulties related to Polish as an IR working language information retrieval algorithm implemented in the Europeana search-engine
As far as the quality of descriptions is concerned Europeana does not bear much responsibility for errors or shortages. Exhibits and documents are indexed independently by their host institutions. Some conclusions or suggestions could, however, help these institutions to modify and unify their description templates. In particular the fields containing the most significant semantic information, such as <dc:subject>, <dc:title> and <dcterms:alternative> should be prepared with care, according to a set of coherent principles. These fields often determine the result of a query and user’s satisfaction.
The second category of conclusions concerns difficulties related to processing of various items and exhibits types. The engine retrieves, amongst others, books, paintings, photographs of architectural and/or natural objects, films, patent descriptions, scientific papers, press items (journals, periodicals), archived administrative documents, or documents of everyday life (e.g. posters, tickets, etc.) Every document type is described according to its own, often specific format, and every country has its own tradition of creating metadescriptions. This variety of local “information cultures” constitutes the richness of European heritage and makes the Europeana project so meaningful. But this is precisely the point where the search-engine encounters the most important obstacle to the effective information retrieval. The question remains open, how to define the reasonable frontier between indexation standards which should remain specific to the country or institution and those which should be imposed as global.
The third set of conclusions concerns linguistic difficulties related to formal,
morphosyntactical and semantic phenomena typical for Polish (as well as for other
Western-Slavic languages). Most of them were described in the Section 2. Formal
constraints of Polish require respecting all the characters of the alphabet, even if some
of them might seem unknown to the users. Morphosyntactical constraints include
Polish flexion which in many cases influences word meaning. Finally semantic
information may be distorted by word homonymy and polysemy. Evaluation process
and quantitative measures obtained in the experiment do not offer clear suggestions
concerning NLP pre-processing of the queries. However, the fact that basic queries
turned out to be more effective than those with manual enrichment is significant (cf.
Table 2). One should stress that in a flectional language like Polish stemming and stop
words removal do not necessarily enhance the effectiveness of retrieval [
Finally, the fourth category of conclusions concerns the information retrieval algorithm implemented in the Europeana search-engine. The results for all the evaluation measures are presented in the Table 2. p-value
Baseline PLWREdu PLWRExp PLWRStu
All the average measures yield the highest values for the basic queries. This means that introducing new terms (enrichment) does not improve the results. Such a conclusion is, at least apparently, counter intuitive. Assuming the correctness of the IR algorithm more information given by a user should result in better scores. Indeed, this would be the case if some terms, such as for instance the first and the last name of a person occurring in one query were connected by conjunction. If all the terms are connected by disjunction, as it was the case, every single term enlarges the number and the scope of documents retrieved, many of them being irrelevant. Reliable comparisons may thus be conducted between queries composed of a similar number of search terms. Actually, when enriched runs are compared alone the results correspond to the expectations, because the highest precision of the retrieved information is attained by expert users and the lowest one by students.
If the MAP measure is taken into consideration the decrease of precision is very high between basic and the other queries, but remains low between the enriched queries PLWREdu, PLWRExp and PLWRStu (Table 3).
Run id Baseline PLWREdu PLWRExp
PLWRStu 5
The experiment shows that working on Polish topics in Europeana is not sufficiently effective and important improvements are necessary in the search engine. There were only 26.6% of highly relevant responses to the queries and 14.8% of partly relevant ones. If partly relevant and irrelevant responses are considered as one set, as much as much as 73.5% of queries were judged by the users as unsatisfactory. Automatic and manually enriched queries were evaluated by users and the results of these evaluations were compared with respect to the Mean Average Precision. MAP produced the best results for the automatic search (0,314), while enriched files contained less relevant results: for the expert users MAP value was 0,1795, for the educated users it was 0,1529 and for students 0,1279.
The results are due to the incoherence of object descriptions and to some linguistic traps of Polish as a retrieval language. An important factor seems to be the algorithm analyzing the entire descriptions without any specification of relevant or irrelevant fields. The problem of sensitivity of the IR systems to the specific Polish characters should be also treated. Finally uniterm indexing strategy, based on matching documents to a single keyword from topic, is not the best choice.
Should Europeana become a truly European project, specific features of the description languages representing a large number of items in the resources ought to be respected, because an integral information system includes also its users and their habits.