For our rst participation to the CLEF CHiC Lab, we submitted runs to the multilingual ad-hoc and multilingual semantic enrichment tasks. Given the strong multilingual character of the evaluation corpus, the main objectives of the experiments were to test the e ciency of semantic topic expansion and consolidation based on Explicit Semantic Analysis (ESA) versions in di erent languages. Another objective was multilingual fusion of results obtained in the di erent languages of the corpus. ESA was adapted for the 10 languages that are best represented in the Europeana corpus. Wikipedia dumps from March 2012 were used for French and English and from March 2013 for the other languages. One problem that arises when modeling short documents, such as queries, with classical ESA vectors no information is available whether the concept is related to the entire topic only to a part of it. To overcome this problem, two adaptations of ESA (ESA-C) are Wikipedia concepts that are linked to the highest number of concepts from the original topic. In the ad-hoc task, ESA and ESA-C have two roles: to expand the topic with related concepts and to create consolidated topic models which contain the original topic words along with other related keywords. We submitted both monolingual and multilingual runs without topic expansion and using topic expansion and consolidation with classical ESA and ESA-C. The best results are obtained in a multilingual setting with no expansion and ESA-C topic consolidation. For the semantic enrichment task, we propose lists of related Wikipedia concepts using either a monolingual ranking or a voting scheme that surfaces related concepts that appear in the largest number of languages. Here, the best results are obtained in a monolingual ranking con guration that exploits ESA-C.
Explicit Semantic Analysis [
Radinsky and al. [
Hassan and Mihalcea [
We proposed an ESA adaptation to information retrieval tasks that gives priority to categorical
information [
ESA has only weak language dependence and was already deployed in several languages. Sorg
and Cimiano [
Put simply, ESA exploits classical text weighting schemes, such a tf-idf, to model concepts from a structured resource, such a Wikipedia. A relation between words and the concepts that structure the space is established by inverting the concepts' vectorial representations. Thus, each word of the vocabulary has an associated high-dimension projection onto the concept space of the underlying resource. Finally, in order to compare two words or two documents, the representations of individual words are summed and the resulting vectors compared. In information retrieval, the most useful component of ESA is the mapping of words onto concepts that can be used for topic expansion or consolidation (see Subsection 2.2.
Classical ESA representations are well adapted for single words, since there is nothing to be done, and for long documents, since the summing operation smooths individual contributions and an accurate semantic representation of the document is obtained. However, the method has some drawbacks for documents such as retrieval topics that contain only few words (typically 2 to 4 words). Here, the smoothing of individual contributions is not su cient because the contribution of a single word can be higher than that of the others and the obtained related concepts could be related to a part of the topic only. An illustration of this type of problem is provided in table 1 which presents the top 10 ESA concepts associated to topics Freshwater Fish and JeanJaques Rousseau3. The results from table 1 indicate that most ESA top ranked concepts are not related to the entire query. When examining results for topic CHiC-051, Freshwater bivalve and Freshwater, Isle of Wight are related to freshwater while Bait sh and Bank shing are related to sh. Similarly, when examining results for topic CHiC-058, we notice that several ESA top concepts are brought up by the family name textitRousseau and have little semantic relatedness 3 The wrong spelling of Rousseau's name was extracted from Europeana logs and provided as such for the task. with the original topic. Concepts found for topic CHiC-064 (crockery doll house) are related to doll but not the other terms from the query.
We use an in-house implementation of ESA that includes only the optimization cues publicly
available until recently 4. To validate our implementation, we performed the word similarity task
described in [
Since topics are often short, ties are often obtained with the boolean score and their are broke using the second, ner grained score.
The introduction of the boolean score has two main objectives. First, categorical information should be favored in order to obtain concepts that are hierarchically related (i.e. isA relation) to the initial topic or to parts of it. Second, it is possible to identify which parts of the initial query an ESA related concept is related to. For instance, the categories of Tropical Fish are Fish stubs and Aquaria and the topic would have a boolean score of 1 (out of a maximum of 2). Similarly, Freshwater bivalve, the top ranked concept with classical ESA, only loosely related to the initial topic, has a boolean score of 0 since its only category is Bivalves. The categorical ranking rightly gives a better position to Tropical Fish compared to Freshwater bivalve since the rst concepts is more closely related to Freshwater sh.
Here we modi ed our ESA adaptation for IR in two directions. First, given that categorical information is often sparse, we added the words contained in the rst 150 characters after the concepts name in the rst paragraph of the category words. This enrichment of the categorical space is motivated by the fact that the rst paragraph of Wikipedia article is often a de nition that contains salient concepts related to the target one. The limitation to words contained in string of 150 character is useful since the rst paragraph has varying length and contains information that is only loosely related to the concepts when it is long. The second modi cation is a concept detection that is used to produce a third score which favors articles that contain longer concepts from the initial query over other articles. At equal categorical scores, the inclusion of concept detection allows us to favor a Wikipedia concept that includes Jean-Jacques Rousseau in its text when compared to another concept that includes Jean-Jacques and Rousseau separately. The top related concepts obtained with ESA-C for Freshwater sh and Jean-Jacques Rousseau are presented in the third column of table 1. In both cases, the top 10 concepts are much more closely related to the initial topics compared to the use of classical ESA. The list for topic CHiC-051 contains only sh and the list for CHiC-058 includes di erent works of Jean-Jacques Rousseau. Crockery doll house is a specialized one, which is not well represented in Wikipedia and the retrieved concepts are still unrelated to the entire topic in a large majority of cases. This last topic illustrates one limitation of all ESA implementation, namely the poor mapping between the initial document and the knowledge included in the underlying conceptual space.
The use of the lists of related ESA concepts for both semantic enrichment and for ad-hoc retrieval is detailed in Section 4. 3
We kept documents of the Europeana Collection that belong to the 10 languages processed with ESA. Separate indexes were created for each of the modeled languages. Then selected metadata associated to the following elds: \dc:title", \dc:description", \dc:subject", \dc:type", \dcterms:medium", "dc:date\. The retained elds were merged into a bag-of-words representation and then modeled using a tf-idf scheme. Due to the use of probabilistic models of the topics (see Subsection 4.2), the tf-idf representation of documents was subsequently transformed into a probabilistic form by dividing the weight of each word by the sum of the scores of all words in the document. Monolingual collections are stemmed using the corresponding Perl Snowball stemmer implementation.
The framework devised here was used for both semantic enrichment and ad-hoc retrieval and is summarized in gure 1. The semantic enrichment process exploits only topic expansion with the ESA versions (ESA and ESA-C) and returns ranked lists of results using di erent ranking schemes detailed in Subsection 4.2. 4.1
The purpose of the semantic enrichment process is to return a ranked list of concepts that are semantically related to the initial topic and could be use for query expansion. To test multilingual rankings, we introduced fusion methods that exploit the explicit interlingual links available in Wikipedia using either di erent fusion schemes based on the scores in individual languages. In all cases, the proposed enrichments are collection independent. Only Wikipedia concepts formed of at most 4 words were retained in the nal rankings. Lists of related concepts obtained with the original version of ESA and with the adapted ESA-C version are presented in table 1. 4.2
Within CHiC, the objective of the ad-hoc retrieval process is to return the best results possible using whatever automatic method at hand. In our approach, the target topic is rst processed using ESA resources to expand and consolidate it. The initial words and the expanded concepts are then compared to the index of the collection in order to retrieve a raw list of results. The elements of this list are then compared to the consolidated version of the topic in order to obtain the nal list of results. Similarly to the ranking of ESA related concepts, two similarity measures are used: { a boolean score to measure a coarse similarity between the initial topic or its related concepts and the documents in the collection. { the cosine similarity is used to measure the degree of similarity between a topic and corresponding documents.
The boolean score has a higher priority than the cosine similarity, which is used only to break ties. For multilingual runs, the process is performed for each of the languages processed and then results are combined by ranking results by decreasing scores.
Topic expansion is performed in a way similar to description provided in Subsection 4.1. Consolidation is a by-product of the expansion process and aims to obtain an expanded version of the initial topic that contains, in addition to the original words, other words that are semantically related to the topic but are not part of it. The words are ranked by summing their individual scores associated to the top 100 ESA related concepts and then by multiplying this sum with the log of the number of di erent articles in which they appear. This last score is used in order to favor words that are associated to a large number of Wikipedia concepts related to the initial articles. Up to 1000 related words are retained for each topic and a probabilistic model of the consolidated versions is obtained by dividing individual word scores by the sum of all scores. In table 2, we present top 10 words related to each topic obtained with ESA and ESA-C. A majority of the obtained words are semantically related to the initial topic, although some outliers appear. For freshwater sh, all top 10 words are related to the initial topic and thus useful for ranking results. In the case of Jean-Jaques Rousseau, there are French stop words that were not removed. For Crockery doll house, nrhp (abbreviation of National Register of Historic Places) appears since this acronym is strongly related to house. Similarly to the collection processing, the consolidated versions of the topics are stemmed. 5
As we mentioned, we have submitted runs for both the semantic enrichment and the ad-hoc retrieval subtasks and we analyse them here. Unfortunately, this analysis is altered by the fact that an important bug in the scoring of related was discovered after the release of o cial results. This bug had a strong negative impact on the quality of results for all runs that exploited fusion techniques for semantic enrichment and automatic topic expansion for ad-hoc retrieval. The bug biases individual boolean scores but the order of concepts is not a ected and a comparison of ESA versions remains possible. Boolean scores of expanded concepts were overrated compared to the boolean scores of documents found using terms from the original topic. All a ected runs are indicated by a "*\ sign in the following subsections. 5.1
Submitted runs The following eight runs were submitted to the semantic enrichment subtask: { ceaListEnglishMonolingual - Related concepts are obtained with ESA-C. The experiment is monolingual since it only exploits the English Wikipedia version. Proposed expansions are collection independent. { ceaListEnglishMonolingualOriginal - Related concepts are obtained with classical ESA. The experiment is monolingual since it only exploits the English Wikipedia version. Proposed expansions are collection independent. { ceaListEnglishRankEnglish - Rank fusion for monolingual results obtained with ceaListEnglishMonolingual. The rank of the concept is obtain by averaging its ranks in di erent languages. For a Wikipedia concept to be considered, it has to appear in at least three languages. The experiment is multilingual since di erent Wikipedia versions (9 languages: en, fr, de, es, it, nl, no, sv, pl) are used. Only concepts that have an English version are considered. { ceaListEnglishRankMultilingual - Rank fusion for monolingual results obtained with ceaListEnglishMonolingual. For a Wikipedia concept to be considered, it has to appear in at least three languages. The experiment is multilingual since di erent Wikipedia versions (9 languages: en, fr, de, es, it, nl, no, sv, pl) are used. Given di erent translations of a concept, the one that has the highest score in an individual language is presented. { ceaListEnglishBooleanEnglish * - Fusion of boolean scores for monolingual results obtained with ceaListEnglishMonolingual. For a Wikipedia concept to be considered, it has to appear in at least three languages. The experiment is multilingual since di erent Wikipedia versions (9 languages: en, fr, de, es, it, nl, no, sv, pl) are used. Only English versions of Wikipedia concepts are presented. Proposed expansions are collection independent. Only concepts that have an English version are considered. { ceaListEnglishBooleanMultilingual * - Fusion of boolean scores monolingual results obtained with ceaListEnglishMonolingual. For a Wikipedia concept to be considered, it has to appear in at least three languages. The experiment is multilingual since di erent Wikipedia versions (9 languages: en, fr, de, es, it, nl, no, sv, pl) are used. Given di erent translations of a concept, the one that has the highest score in an individual language is presented. Only concepts that have an English version are considered. { ceaListEnglishCosineEnglish * - Fusion of cosine similarity scores for monolingual results obtained with ceaListEnglishMonolingualOriginal. For a Wikipedia concept to be considered, it has to appear in at least three languages. The experiment is multilingual since di erent Wikipedia versions (9 languages: en, fr, de, es, it, nl, no, sv, pl) are used. Only English versions of Wikipedia concepts are presented. Only concepts that have an English version are considered. { ceaListEnglishCosineMultilingual * - Fusion of cosine similarity scores monolingual results obtained with ceaListEnglishMonolingualOriginal. For a Wikipedia concept to be considered, it has to appear in at least three languages. The experiment is multilingual since di erent Wikipedia versions (9 languages: en, fr, de, es, it, nl, no, sv, pl) are used. Given di erent translations of a concept, the one that has the highest score is presented. Only concepts that have an English version are considered.
Results Even though the results for 6 out of 8 runs are biased here, there are some interesting conclusions that we can draw from table 3. The comparison between ceaListEnglishMonolingualOriginal and ceaListEnglishMonolingual is favorable to the latter method. The original ESA implementation has signi cantly poorer performances compared to the adapted method introduced (P@10 0.365 vs. 0.66). The privileged role given to categories and to the rst words in the concept text, coupled with concept detection in the queries have a positive impact on semantic enrichments.
None of the fusion methods proposed improves results compared to the best submitted run but this is at least in part due to the bug that a ected the values of boolean concept scores. When comparing the fusion schemes, there are no signi cant di erences between monolingual and multilingual fusions. Since the same concepts were proposed but languages di ered, this results show that the ground truth is of high quality. The cosine-based fusion strongly degrades results, while the fusion based on ranks is closer to the original results.
Important di erences occur at the topic level. For ceaListEnglishMonolingual, when examining CHiC-51 (freshwater sh) and CHiC-58 (Jean-Jacques Rousseau), all top 10 related concepts are at least partly related to the initial topic. Inversely, results are very poor (9 out of 10 irrelevant enrichments) for topics CHiC-64 (crockery doll houses ) and CHiC-65 (sea sunset ). These failures are probably due to a poor mapping of the topic in the Wikipedia corpus for CHiC-64 and to the very small number of Wikipedia concepts that cover both sea and sunset. 5.2
Submitted runs For "noExpansion\ runs, results are ranked rst by the number of terms from the initial topic that appear in the document and then by the the cosine similarity between the consolidated version of the topic and document representations. For the other runs, the boolean score of related ESA concepts biases the results. As we mentioned, multilingual fusion was performed by The following 16 runs were submitted to the semantic enrichment subtask: { ceaListMultilingualNoExpansion - Multilingual run that retrieves only documents which contain at least one word from the initial topic. { ceaListFrenchNoExpansion - Monolingual French run that retrieves documents which contain at least one word from the initial topic. { ceaListGermanNoExpansion - Monolingual German run that retrieves documents which contain at least one word from the initial topic. { ceaListMultilingualOriginal * - Multilingual run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Wikipedia concepts obtained with classical ESA. { ceaListMultilingualFiltered * - Multilingual run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Wikipedia concepts obtained with ESA-C. { ceaListDutchFiltered * - Monolingual Dutch run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Dutch Wikipedia concepts obtained with a version of Explicit Semantic Analysis adapted to short documents (topics). { ceaListEnglishFiltered * - Monolingual English run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related English Wikipedia concepts obtained with ESA-C. { ceaListFrenchFiltered * - Monolingual French run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related French Wikipedia concepts obtained with ESA-C. { ceaListGermanFiltered * - Monolingual German run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related German Wikipedia concepts obtained with ESA-C. { ceaListItalianFiltered * - Monolingual Italian run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Italian Wikipedia concepts obtained with ESA-C. { ceaListNorwegianFiltered * - Monolingual Norwegian run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Norwegian Wikipedia concepts obtained with ESA-C. { ceaListPolishFiltered * - Monolingual Polish run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Polish Wikipedia concepts obtained with ESA-C. { ceaListSpanishFiltered * - Monolingual Spanish run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Spanish Wikipedia concepts obtained with ESA-C. { ceaListSwedishFiltered * - Monolingual Swedish run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Swedish Wikipedia concepts obtained with ESA-C. { ceaListEnglishOriginal * - Monolingual English run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related English Wikipedia concepts obtained with ESA-C. { ceaListItalianOriginal * - Monolingual Italian run that retrieves documents which contain at least one word from the initial topic and/or the full name of one of query's 1000 most related Italian Wikipedia concepts obtained with classical ESA.
Results Due to the bug that a ected all the runs that involved ESA based topic expansion, it is di cult to compare runs that did not involved expansion and the others. However, it is worthwhile noticing the the best submitted run, i.e. ceaListMultilingualNoExpansion a simple fusion of results obtained for individual languages, gave interesting results compared to monolingual runs that involved no ESA expansion.
When comparing ceaListMultilingualOriginal and ceaListMultilingualFiltered, the two
multilingual runs that exploit ESA and ESA-C, obtained results are better for the second run (MAP
0.0805 vs. O.0977). This result con rms the one obtained for semantic enrichment, where ESA-C
was also superior to classical ESA. It is also in line with our ndings from [
Future work includes adding supplementary material to the conceptual space (i.e. Wikipedia corpus) in order to enrich concept descriptions and to try to cover a larger range of concepts. A second line of work involves a shift towards a more semantic representation of ESA concepts, that goes beyond the current bag-of-words modeling and involves concept detection and disambiguation. A third research axis will focus on ways to predict the chances of success of automatic expansion in order to perform this task only when the topic is suited.
This work was supported via Egonomy, a French "Grand Emprunt\ project.