The Cultural Heritage in CLEF 2012 (CHiC) pilot evaluation included these tasks: ad-hoc retrieval, semantic enrichment and variability tasks. At CHiC 2012, the University of She eld and the University of the Basque Country submitted a joint entry, attempting the three English monolingual tasks. For the ad-hoc task, the baseline approach used the Indri Search engine. Query expansion approaches used random walks using Personalised Page Rank over graphs constructed from Wikipedia and WordNet, and also by nding similar articles within Wikipedia. For the semantic enrichment task, random walks using Personalised Page Rank were again used. Additionally links to Wikipedia were added and further approaches used this information to nd enrichment terms. Finally for the variability task, TF-IDF scores were calculated from text and meta-data elds. The nal results were selected using MMR (Maximal Marginal Relevance) and cosine similarity.
The Cultural Heritage in CLEF 2012 (CHiC) pilot evaluation proposed these tasks: ad-hoc retrieval, semantic enrichment and variability tasks.
The University of She eld and the University of the Basque Country submitted a joint entry, attempting the three English monolingual tasks. are based on real Europeana3 query logs and the documents to be retrieved are metadata records of Europeana objects.
We participated in the English monolingual subtask and submitted 4 di erent runs: one baseline run, and other 3 runs applying query expansion.
For all our approaches, we used Indri search engine [
As we did not have training data, we xed the parameters to the optimum values in other previous experiments (w = 0:7).
We use two di erent approaches to obtain the expansion terms.
Using Random Walks. The query expansion algorithm based on random walks over the graph representation of concepts and relations in a knowledge base to obtain concepts related to the queries.
We have use this approach for two di erent runs. The di erence between
these two runs is the knowledge-base used. We have used Wikipedia for one
run (EXP UKB WIKI10 is the identi er for this run), and WordNet [
relations provided, including the gloss relations. This was the setting obtaining
the best results in a word similarity dataset as reported by [
Given a query and the graph-based representation of Wikipedia or WordNet,
we obtain a ranked list of related concepts as follows:
1. We rst pre-process the query to obtain the lemmas and parts of speech of
the open category words.
2. We then assign a uniform probability distribution to the terms found in the
query. The rest of nodes are initialized to zero.
3. We compute personalized PageRank [
Basically, personalized PageRank is computed by modifying the random jump distribution vector in the traditional PageRank equation. In our case, we concentrate all probability mass in the concepts corresponding to the words in the query.
Let G be a graph with N vertices v1; : : : ; vN and di be the outdegree of node i; let M be a N N transition probability matrix, where Mji = d1i if a link from i to j exists, and zero otherwise. Then, the calculation of the PageRank vector Pr over G is equivalent to resolving Equation (2).
Pr = cM Pr + (1 c)v (2)
In the equation, v is a N 1 vector and c is the so called damping factor, a scalar value between 0 and 1. The rst term of the sum on the equation models the voting scheme described in the beginning of the section. The second term represents, loosely speaking, the probability of a surfer randomly jumping to any node, e.g. without following any paths on the graph. The damping factor, usually set in the [0:85::0:95] range, models the way in which these two terms are combined at each step.
The second term on Eq. (2) can also be seen as a smoothing factor that makes any graph ful ll the property of being aperiodic and irreducible, and thus guarantees that PageRank calculation converges to a unique stationary distribution.
In the traditional PageRank formulation the vector v is a stochastic normalized vector whose element values are all N1 , thus assigning equal probabilities to all nodes in the graph in case of random jumps. In the case of personalized PageRank as used here, v is initialized with uniform probabilities for the terms in the document, and 0 for the rest of terms.
PageRank is actually calculated by applying an iterative algorithm which computes Eq. (2) successively until a xed number of iterations are executed. In our case, we used a publicly available implementation5.
In order to select the expansion terms, we choose the top N highest scoring concepts. When using Wikipedia, the rst 10 concepts are used as expansion
terms. In the case of WordNet, we get all the words that lexicalize the rst 100 concepts.
For instance, given a query like \Esperanto", this method based on Wikipedia suggests related terms or phrases like L. L. Zamenhof, interlingua, international auxiliary language and constructed language.
Using Wikipedia Similarities. For the second query expansion approach, we use the 10 concepts obtained by the WIKISIM approach for the Semantic Enrichment task (see Sec. 4). We refer to this query expansion approach as the EXP SE WIKISIM run. 2.3
Method MAP EXP UKB WN100 51.61
NOEXP 51.48 EXP SE WIKISIM 50.96
EXP UKB WIKI10 50.64
The approaches from our submission give the best results overall in comparison to the other submissions. The baseline NOEXP approach provides strong results. However the query expansion approaches give little improvement, or even slightly degrade performance. 3
The goal of this task is to present a list of 12 items that give a good overview
[
Indexing and searching of the collection was performed using Apache Solr6. By default only the dc:title, dc:description, and dc:subject elds were searched and all words in the query were required to appear in those elds. Basic fully automatic query expansion was used to achieve higher recall. For singular nouns the plural form was added as a search keyword, vice-versa for plural nouns the singular form was added. If a word looked like a year, then the enrich:period label and europeana:year elds were also searched for that year. Additionally we used
a very small gazetteer of place-names to identify candidate toponyms which were then searched for in the enrich:place label eld.
For each of the query result documents we then calculated two TFIDF scores,
one based on the textual description of the items (labelled TEXT, using dc:title,
dc:description, and dc:subject elds), the other based on what we termed the
meta-data facets (labelled FACET, using dc:subject, europeana:dataProvider,
enrich:place label, europeana:year, europeana:type, and dcterms:medium elds). For
the textual descriptions the dc:title and dc:description were sentence and word
tokenised, and then stop-worded using NLTK7. For all other elds the whole
eld content was used as a single token. Using the TFIDF scores the nal 12
documents were then selected using either Maximal-Marginal-Relevance (MMR)
[
Maximal-Marginal-Relevance The selection using MMR starts by clustering
the documents using k-means. The number of clusters k is selected
automatically [
Cosine Similarity In the cosine similarity method we randomly select a document to be the rst document. The remaining documents are then sorted by decreasing cosine similarity to the rst document. From this ranking we then sample the nal 12 documents at regular intervals. 3.2
Method MAP CLUSTERFACETS 23.93 CLUSTERTEXT 23.13 SIMFACETS 22.59
SIMTEXT 21.85
The results from our submission give the best results overall when compared to the other submissions. As the variability judgements have not yet been released, the quality of the results is hard to judge. However the results seem to
imply that the cosine similarity produces better results if there are about 12 topics in the results and the topics have roughly the same number of items each, whereas the MMR method works better if this condition does not hold. The most likely reason for this is that the regular sampling used in the cosine similarity method breaks if the topic distribution is heavily skewed, or there are only a few topics. 4
The goal of this task is to present a ranked list of at most 10 concepts (words or phrases) for each query. These concepts should semantically enrich the query and/or guess the information need or query intent of the user. The concepts can be extracted from the Europeana data that has been provided (internal) or make use of other resources such as the Web or Wikipedia (external). 4.1
Random walks The concepts from the UKBWIKI and UKBWN runs are obtained using Wikipedia and WordNet, respectively, following the expansion strategy based on random walks explained in Section 2.2.
Wikipedia links Two approaches attempted to nd useful terms by nding inline Wikipedia links within each of the items in the collection. So for example given the text: Hiroshima peace lanterns at Leith 1985.
Leith; Ceremonies; Peace demonstrations; Eighties; War in Japan; World War II keywords
Links might be added to the terms Hiroshima, Leith and World War II. The motivation behind this approach is that these added links will suggest useful keywords which co-occur with the query term and so could be used as semantic enrichments for the term.
The Wikipedia Miner software [
The rst approach used an IR engine to nd items that contained the query term. The IR engine used was Apache Solr (as described in Section 3). All returned items were then run through Wikipedia Miner to markup Wikipedia links. A con dence threshold of 0.2 was used to eliminate low con dence links. The links that most often co-occurred within the items were then returned as the semantic enrichments. So for example World War II occurred frequently as a link in items containing `Hiroshima', and so was returned as a enrichment term. This method is referred to as QUERYLINKS.
The second approach used a slightly di erent method. Here a previously processed version of Europeana was used which had already been run through Wikipedia Miner to nd links for all items. Then instead of searching for items containing the query term using the IR engine, items that contained a link to the query term were used instead. So for example instead of searching for `hiroshima', only items that contained a link to `Hiroshima' were used. As with the rst approach the most frequently co-occuring links were returned as the enrichment terms. One problem with this method is that sometimes very few or no items were found which contained a link to the query term. So as a fallback, a di erent method was used which made reference only to Wikipedia. The method found articles that were most similar to the query term, by examining the in and outlinks to and from the article and returning the article titles which contained the highest proportion of these in/outlinks. This method is referred to as WIKISIM. 4.2
Method MAP UKBWIKI 29.05 UKBWN 20.70 WIKISIM 19.29
QUERYLINKS 16.61
The UKBWIKI was the strongest run from our submission, showing the effectiveness of the random walk approach. The di erence between the UKBWIKI and UKBWN results shows that the richness of Wikipedia provides a substantial bene t over WordNet. The WIKISIM and QUERYLINKS approach come around the middle of the table, with WIKISIM faring substantially better. This would seem to show that returning similar articles based on links seems to be quite e ective for this task. 5
For the ad-hoc retrieval task, a strong baseline approach achieved better results than all the other submissions. The query expansion approaches presented here did not improve substantially on this performance.
Our submission for the variability task achieved the best results overall using text and facet similarity calculations to select the items to return. The MMR clustering approach proved slightly more e ective than the cosine similarity measure.
For the semantic enrichment task the random walk approach proved e ective, giving the 3rd and 6th overall highest precisions. The richer graph produced from Wikipedia proved to give substantially better results than using WordNet. The Wikipedia link approach gave results about midway in the table. Finding similar articles (in the WIKISIM run) proved to be slightly more e ective than using the links alone (QUERYLINKS).
The research leading to these results was carried out as part of the PATHS project (http://paths-project.eu) funded by the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 270082 and KNOW2 project (TIN2009-14715-C04-01).