We will report on the participation of GESIS at the first CHiC workshop (Cultural Heritage in CLEF). Being held for the first time, no prior experience with the new data set, a document dump of Europeana with ca. 23 million documents, exists. The most prominent issues that arose from pretests with this test collection were the very unspecific topics and sparse document representations. Only half of the topics (26/50) contained a description and the titles were usually short with just around two words. Therefore we focused on three different term suggestion and query expansion mechanisms to surpass the sparse topical description. We used two methods that build on concept extraction from Wikipedia and on a method that applied co-occurrence statistics on the available Europeana corpus. In the following paper we will present the approaches and preliminary results from their assessments.
In 2011 the CHiC workshop (Cultural Heritage in CLEF) was held for the first time at CLEF. After this initial workshop, where first ideas on how to evaluate the retrieval of “cultural assets” were discussed, CHiC started as a pilot evaluation lab in 2012. Therefore we neither had prior experience with the data set that was provided by the lab organizers nor with the specific domain (cultural heritage) in particular. To allow a systematic IR evaluation the organizers published a dump from the Europeana platform with roughly 23 million documents to form the initial evaluation corpus and a total of 50 different topics that were used in three different tasks. GESIS contributed in two of these tasks: the Ad-hoc Retrieval Task and the Semantic Enrichment Task. In the following paper we will present our approaches and preliminary results from their assessments.
Since this lab had no predecessor we couldn’t learn from previous results and best practices. So, the main objective of this initial participation was to establish the retrieval environment, to get a feeling for the data set and to surpass the obvious issues in the first place.
The 50 topics provided were very unspecific and underspecified. They consist of the usual identifier, title and description and were provided in three languages (English, French, and German). Only half of them (26/50) contained a description and the titles were usually short with just around two words (see Table 1). As stated by the official lab guideline, the topics “are taken from real Europeana query logs and comprise queries for people, places, work titles (e.g. Mona Lisa), events or subjects”. This might explain the sparse representation of the topics, but it never the less is a source of serious retrieval issues that led to the main objectives of our experiments.
When using unprocessed queries on all available metadata fields some topics just produced very small result sets, e.g. topic CHIC-009 (“falkland islands”) only returned 12 documents at all. So, we decided to focus on the Semantic Enrichment Task to acquire a rich set of additional query terms that can be used for the necessary query expansion mechanisms in the Ad-hoc Retrieval Task.
In the following paper we will first present our overall technical retrieval system, different filters, and modifiers we used to access the raw data. We will then continue by outlining three different approaches to find appropriate terms for our query expansion. We will conclude with a discussion of the performance of the different approaches and the first lessons learned. 2
We choose the open-source search framework Solr1 to index and query the Europeana dump provided by the CHiC organizers. Within Solr it is easy to import the XML data for the evaluation without extensive schema conversions. A main idea behind using Solr in a stand-alone configuration was that the original Europeana platform also uses this technology stack. At GESIS we have made good experiences with Solr in a number of projects like the social science information portal Sowiport2. We used the Solr 4 nightly build (build no. 4.0-2012-05-29_08-19-37) for indexing and querying. At the time of the evaluation it was not a stable release but it offered some benefits, like included information analyzers that we used in the evaluation.
2.1
To keep the configuration and schema simple, we used the original Solr configuration and imported the Europeana dump via dynamic fields into the provided schema.xml. Using dynamic fields we stored the information as they are delivered in strings and also for each language in a language based field type. The fieldnames remained as in the original description. For English language information we used the available "text_en" field type, for German language information the "text_de" fieldtype.
The available language specific field types in Solr 4 offer the following analyzers3 for all languages: StandardTokenizerFactory: A general purpose tokenizer, which divides a string into tokens with various types. StopFilterFactory: Words from the Solr included stopword lists are discarded. LowerCaseFilterFactory: All letters are indexed and queried as lowercase. Additionally, language specific analyzers4 were used, for English and German: EnglishPossessiveFilterFactory, PorterStemFilterFactory: A stemmer for English, GermanNormalizationFilterFactory, GermanLightStemFilterFactory: A stemmer for German.
With the use of a copyField we stored all separate field information in a common search field (chic-all). 2.2
After indexing the Europeana information in Solr, we were able to create an overview about the coverage of different fields for English and German (see Table 2). We see that the “europeana” namespace is nearly completely available for all datasets. The basic namespace, that includes many available information is the Dublin Core (“dc”) namespace. After that follows the “enrichment” namespace and rarely filled are fields from the “dcterms” namespace. In our evaluation we will concentrate our queries to data fields that are mostly available. 3
We introduce three different techniques to acquire related concepts that would later be used to allow a query expansion. We tried to find related concepts in Wikipedia outlinks from the lead section (Section 3.1), from the Wikipedia full text using document similaritiy (Section 3.2) and the Europeana corpus itself using co-occurrence analyses (Section 3.3).
In this approach we use related concepts from Wikipedia summaries to semantically enrich CHIC topics. Wikipedia articles represent important concepts of the world knowledge. If we can find a Wikipedia article that represents the topic, we can use the text of the article to extract important concepts that are related to it. In this implementation we used links from the lead section, which summarizes the whole article with the most important aspects. Wikipedia guidelines for the lead section5 suggest: “The lead should be able to stand alone as a concise overview. It should define the topic, establish context, explain why the topic is notable, and summarize the most important points […]”. Therefore it could be well-suited to find nearby related concepts. Links in the summary represent relations to other important concepts existing in Wikipedia, which we use for the semantic enrichment.
The approach therefore consists of two steps: (1) finding an appropriate Wikipedia article that represents the topic and (2) extracting links from the article’s first paragraph as a representation of important concepts.
For the first step, we have created a SOLR index of all titles of Wikipedia articles. In several iterations we search for the topic with the (a) original topic, (b) the topic excluding stop words, (c) the permutation of topic words and (d) individual words from the topic. The title with the highest TF*IDF score is then used as a representation. For nearly all topics we were able to find a Wikipedia article that represents it. Problems occur with concepts not contained in Wikipedia (like topic CHIC-049 “teufelstal”), very broad topics (like topic CHIC-020 “europa maps 1914”) or topics that must be searched not only by the title, but in the full text (topic CHIC-037 “1809 combat”)
In the next step we extracted all links from the summary of the article. Therefore, we first took the original WikiSyntax from the article through the Wikipedia API. Then, we clean the article text from internal/system links and other sections like info boxes. As a next step, we extract the summary above the first header. Links from this section are then extracted with regular expressions, utilizing the fact that they are marked with double square brackets. If we do not get enough links from this, we use the whole article text, for example for very short articles. The extracted links are then used as semantic enrichments for the original topic. 3.2
using
In this second approach we use Wikipedia full texts to extract related concepts. For our training corpus we use a particular subset of Wikipedia. In contrast to the previous approach we map Wikipedia entries of the given terms manually.
We used two different Wikipedia training corpora to enrich the query terms. The
first corpus is the German Wikipedia corpus that consists of 1,054,842 articles. The
articles are here randomly selected and the corpus comprises almost the half of the
5 http://en.wikipedia.org/wiki/Wikipedia:Manual_of_Style_%28lead_section%29
complete German Wikipedia corpus6. The second corpus contains only articles having
backlinks or outlinks to the particular English Wikipedia entries taken from the 50
CHIC query terms. We extracted the back- and outlinks using the RelExAPI tool7
(also used [
As an initial step, we index each training corpus using Lucene8. Since each term is represented as an article, given a query term, the related terms are the articles that are most similar to the source. Therefore, this task can also be seen as finding related documents. Formally, we define the similarity score of two documents, denoted as d1 and d2, as follows ( ) | | where d1 and d2 are vectors of words with cardinality of n. We set a factor n to denote the number of important words included in the calculation. The importance of a word can be obtained by calculating its TF*IDF score. This method is a slight modification of the Jaccard similarity coefficient. 3.3
A common approach to find related concepts is the use of co-occurrence analysis. It is
presented extensively in standard natural language processing handbooks e.g. the one
by Manning and Schütze [
In order to specify which terms are likely to co-occur a similarity measure has to be defined. In our approach we choose the Jaccard Index which is calculated for two attributes x and y where DSx and DSy are two document sets with DSx containing documents with attribute x and DSy with attribute y, respectively. DSxy is the documents set containing attributes x and y (i.e. the intersection of DSx and DSy). The document frequencies dfx, dfy and the collocation frequency dfxy are defined as the size of DSx, DSy and DSxy. The Jaccard similarity is given in the next equation. ( ) | | | | df df df df To cope with large differences in the size of DSx and DSy we modified the index by taking the logarithm.
With this measure we processed the following attributes from the Europeana corpus. Terms appearing in dc:title and/or dc:description were treated as input (i.e. query terms) and their co-occurrence with terms from the fields dc:subject and enrichment:concept_label was measured using the Jaccard Index. To make the process easier to understand let us look at the following example. Topic CHIC-010 consists of the query “film canada”. Therefore, we measured which concepts from dc:subject and enrichment:concept_label co-occurred the most with the terms “film” and “Canada” in title or description. The resulting top 3 concepts are: “poster”, “Cinema and Theatre”, and “popular media”. These concepts appear to be semantically related to the query. However, it also becomes clear that the quality of the related concepts can only be as good as the quality of the vocabulary used in dc:subject and enrichment:concept_label. 4
Each method listed in Section 3 returned a list of ten concepts that we used as the result sets for the Semantic Enrichment Task and to establish a rudimentary query expansion mechanism for the Ad-hoc Retrieval Task. Using the previously described methods we established the following concept extraction services (the abbreviations are the same used for the names of the runs and in the DIRECT system): WIKI_ENTITY – Concepts from the Wikipedia lead section, extracted by detecting outlinks (Section 3.1). WIKI_SIM – Concepts from the Wikipedia full text, extracted from the German
Wikipedia subset (1 million documents) using document similaritiy (Section 3.2). WIKI_BACK – Concepts from the Wikipedia full text, extracted from a Wikipedia subset using document similarities. This subset consists of all back- and outlinks of a given seed document (Section 3.2). STR – Concepts from the Europeana data set, extracted using co-occurrence analyses. The analyses were language dependent, so only the specific language corpus was used (Section 3.3). COMBO – A mixture of all available concepts from the four previous services The result sets for the Semantic Enrichment task were constructed from the top 10 ranked results from the previously described approaches. While the co-occurrence and Wikipedia document similarity approaches returned a ranked list based on the similarity scores, the Wikipedia concept extraction from the lead sections did not include such a score. We used the implicit ranking due to order of their appearance in the text. We suppose that the earlier a linked concept appears in the summary the more important is has to be.
The query construction for the Ad-hoc Retrieval Task was done by taking the original title, removing stopwords and adding the concepts by OR-ing them with the title terms. We boosted the title term by factor 2 (^2) and searched in all available metadata fields in the current language (chic-all, see Section 2). The expanded query for topic CHIC-012 (“moby dick”) and the concepts gained from the Wikipedia lead sections therefore looks like the following example: chic_all-en:(moby OR dick)^2 OR chic_all-en:("Herman Melville" OR "English language" OR "Adventure novel" OR "Sea story" OR "Richard Bentley" OR "Harper Brothers" OR "He rman Melville" OR "The Great American Novel" OR "literature" OR "Ishmael (Moby-Dick)") The systems and concepts used for the expansion are identical to the ones from the Semantic Enrichment Task. Additionally a combination of all concepts was submitted (combo). The different combinations and systems used for each tasks and the results we submitted for the Ad-hoc Retrieval and the Semantic Enrichtment Task are listed in Table 3. All queries are stemmed and pre-processed at query time by the Solr filters that are listed and described in Section 2. 5
In the following section we report on the results from the different implementations presented in the previous sections. As we participated in two tasks we will present the results according to each task. A summary of the results is given in Table 4 and 5. Due to a problem in the DIRECT evaluation system we could not access all data. The figures are based on our own calculations using trec_eval, while the tables are based on the data included in the figures provided by DIRECT. 5.1
Out of the eight system runs we submitted the WIKI implementations could generally produce the most effective results that were above the average performance of all competing systems (AVG_ALL) which was 0.4255 for the English sub-task and 0.5111 for the German. The WIKI_ENTITY system was the best among our systems with MAP value of 0.4396 (EN) and 0.5680 (DE). In both cases the WIKI_ENTITY was among the top 5 participating systems.
In the English sub-task 14-15 topics didn’t returned any documents which resulted in an empty result set and a MAP value of 0. In the German sub-task only 4-6 topics had this problem. Some other topics nearly produced the same MAP value (like CHIC-001, CHIC-006, CHIC-014 or CHIC-034) which might be seen as an indicator that the terms used to expand the query didn’t have any effect.
COMBO_EN
STR_EN
WIKI_BACK_EN
WIKI_ENTITY_EN 1,0 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0,0 1,0 0,9 0,8 0,7 0,6 0,5 0,4 0,3 0,2 0,1 0,0
I-001CH I-003CH I-005CH I-007CH I-009CH I-011CH I-103CH I-105CH I-017CH I-019CH I-201CH I-203CH I-205CH I-027CH I-029CH I-031CH I-033CH I-305CH I-037CH I-039CH I-041CH I-043CH I-045CH I-047C I-049CH C C C C C C C C C C C C C C C C C C C C C C C CH C COMBO_DE
STR_DE
WIKI_ENTITY_DE
WIKI_SIM_DE I-100CH I-030CH I-050CH I-700CH I-009CH I-101CH I-013CH I-501CH I-017CH I-019CH I-102CH I-302CH I-205CH I-270CH I-209CH I-310CH I-033CH I-053CH I-730CH I-930CH I-041CH I-430CH I-405CH I-074C I-904CH C C C C C C C C C C C C C C C C C C C C C C C CH C Fig. 1. Plot of MAP values for the English and German monolingual Ad-hoc Retrieval Task. Out of the seven runs that we submitted to the Semantic Enrichment task, the WIKI_ENTITY implementation could outperform both our own and the competing implementations from other groups. In the manual assessment this approach could achieve a precision of 0.9240 (weak)/0.7000 (strong) in the English monolingual run, and 0.8794 (weak), 0.7448 (strong) in the German monolingual run (see Table 5). All other implementations were below the average precision over all runs. While for the English run the other both WIKI systems could achieve precision values that were only slightly below the average, the STR system could only provide useful enrichments in 1/5 (weak) or 1/10 (strong) of the cases.
When evaluated as a Query Expansion mechanism by the CHiC organizers, the WIKI_ENTITIY system was still among the top 5 systems with MAP values of 0.2338 (EN) and 0.3192 (DE). Surprisingly the reference implementation ORIGINALQUERIESEN by the organizers was the best systems with MAP of 0.3411 (EN) and 0.5701 (DE). Our system outperformed the others in topics CHIC001 and CHIC-017, while it was significantly worse in CHIC-005. Since the implementation details are not clear9, we cannot describe this any further. 6
In this paper we described three different approaches that were implemented in five different systems (see Table 3).
For both Wikipedia methods the mapping of the topic title to a Wikipedia
document were an essential first step. We surpassed this by using a rather ad-hoc
implementation using a separate Solr index (WIKI_ENTITY) or by mapping them by hand
(WIKI_SIM and WIKI_BACK). We are aware of the fact that there are public APIs
to access the Wikipedia content in a more convenient way, like JWPL [
The performance of the STR was worst in all tasks and did not perform very well compared with most other approaches. A possible explanation for this is the lack of consistent controlled vocabulary in Europeana documents and the fact that in at least 50% of all cases no entry was made (see Table 2) in the fields used for co-occurrence analysis (dc:subject and enrichment:concept_label). Looking at the concepts suggested by our service one might also doubt the consistency of the vocabulary used in Europeana. Topic CHIC-019 might serve as an example. The topics title is philosophical anthropology and the resulting concepts suggested by our STR service are: звук, sunet, ήχος, sonido, dźwięk, garsas, zvuk, lyd, ääni, and suono. While the meaning of these recommendations is not always clear it is apparent that they are of the wrong language. These were retrieved from the English dataset and should thus be of English language.
These inconsistencies are also reflected in the results of the assessment campaign. While our approach does perform better than some others it is also clearly surpassed by approaches which use more controlled vocabulary (e.g. our Wikipedia entities approach). Extracting concepts using co-occurrence analysis works well if the given dataset is of high and consistent quality and uses a controlled vocabulary on the majority of its entries. However, in the given case of Europeana approaches that make use of external knowledge are better suited.
The WIKI_BACK system in the Ad-hoc task could produce quite satisfying results that were just under the WIKI_ENTITY system. For the query terms that have Wikipedia entries, this method provides in some cases reasonable results. In contrast, if 9 “[…] we used a Lucene index to compare runs with just the original queries (those runs are marked with ORIGINALQUERIES in the title) to runs that included the original queries plus the semantic enrichment suggested by your experiments. These runs were now assessed for relevance exactly like the ad-hoc runs and you can compare your results with all the standard metrics.” (excerpt from an organizer’s email) the query terms are very general or do not exist as Wikipedia entries, this method would provide “erratic” results. The query terms, e.g. “zeppelin 1900”, “england cup final”, “unarmed”, or “europe maps 1914”, are difficult to be mapped into Wikipedia entries. We have no prior knowledge of what users are actually looking for and we cannot simply adjust these terms into particular Wikipedia entries. As a consequent, we deal with uncertainty in this matter in order to enrich the query terms with Wikipedia and therefore the results provided by this method are rather ambiguous.
In general we could establish a retrieval environment that while being technically comparable with the Europeana (Solr-based) system, surpasses some of the previously described problem in respect to the sparse document and topic representations. Never the less a lot of issues remain unsolved.
Acknowledgements. This work was partially supported by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), grant number SU 647/5-2.