Identifying the most suitable stemmer for the CHiC multilingual ad-hoc task

Abstract

Because the 2013 Cultural Heritage in CLEF (CHiC) lab focused on multilingual retrieval, our goals were the integration of Apache Solr in our Xtrieval framework and the evaluation of di erent stemmers available for most of the relevant languages. As there were thirteen languages to cover, we tried to nd a generic stemmer which works with all languages. We experimented with four setups, where one setup was without any stemmer, two setups used mainly rule-based stemmers and the last setup used a dictionary-based stemmer. For the dictionary-based stemmer we employed the HunSpell stemmer, which works with the same dictionaries as OpenO ce.

Keywords

• stemmer
• evaluation
• dictionary-based stemmer
• rule-based stemmer
• cultural heritage

Introduction

In 2013 the Cultural Heritage in CLEF (CHiC) lab focused on multilingual retrieval, i.e. searching over all available languages. For this task topics and relevance assessments were provided for a total of thirteen languages. This set allowed us to perform a large scale evaluation of stemming algorithms for all available languages.

Our goals for this year were the integration of Apache Solr1 in our Xtrieval framework [ 1 ] and the evaluation of the di erent stemmers available within Solr for a broad use on multilingual corpora. Hence we focused on the multilanguage ad-hoc task.

System overview and setup

As usual we used our Xtrieval framework to carry out all our experiments. But this year we added a new retrieval engine: Apache Solr. It is a very popular open source enterprise search, [ 2 ] which is built on top of Apache Lucene2, and provides several web service interfaces to conduct the di erent tasks necessary to perform large scale searches. 1 http://lucene.apache.org/solr 2 http://lucene.apache.org/java

Although we have already been using Apache Lucene for our experiments since 2006 [ 3 ], this new approach has changed the way we have to con gure them. A considerable amount of con guration is now done in Solr using various XML les. Within these con guration les one can de ne elds with associated types where the processing steps are declared.

CHiC documents

Xtrieval framework no stemmer standard stemmer less aggressive stemmer HunSpell stemmer + stopword filter

Apache Solr

CHiC ad-hoc topic results in TREC format As shown in gure 1, the Xtrieval framework was still an essential part of our experiments. We used it to read and parse the document collection and fed it into the Solr web service. At this point there are still some pre-processing steps left to the Xtrieval framework, which cannot be con gured in Solr.

After the indexing, we used our framework to transform the topics into search queries for the Solr web service and gathered the results in the TREC format in order to submit them.

This year we focused on the di erent stemming approaches, which are o ered by Solr. As there were thirteen languages available, we tried to nd a generic stemmer which covers all these languages. We experimented with four settings: one was without any stemmer, two setups used mainly rule-based stemmers and the last setup used a dictionary-based stemmer. For the dictionary-based stemmer we employed the HunSpell stemmer [ 4 ], which works with the same dictionaries as OpenO ce and LibreO ce. Because of the open source nature of these applications there are large numbers of dictionaries available for almost every language.

Finally, we set up these stemming experiments: { No stemmer { The standard stemmer con gured in Solr: In most cases this is the snowball stemmer or another rule-based stemmer. Where no other stemming algorithm was available the HunSpell stemmer was used. { A less aggressive stemmer: This was used instead of the standard stemmer for every language where a less aggressive stemmer was available. { The HunSpell stemmer: Only languages with HunSpell dictionaries which had a reasonable performance used this stemmer. There were performance problems with complex languages, like for example French, where probably many connections exist between dictionary entries. The bad performance for this language sometimes resulted in a processing time 1000 times longer than with the standard stemmer. Therefore we used no stemmer for these languages.

Furthermore we applied a language detection to determine elds with wrongly labeled languages and assigned the detected language. We also removed stop words for each language. Each language was indexed into the same index, but using separate elds. When processing the topics for each available language a query was issued to the whole index, i.e. every language in the index.

The mapping of the XML data to the elds of the Solr/Lucene index was the same we used for our experiments last year. [ 5 ] 3

Results

The di erence between standard and less aggressive stemming is marginal and cannot be rated as signi cant. Without any stemmer, the results are the lowest and the HunSpell stemmer performed slightly better than no stemming at all. 4

Conclusions and future work

It is evident that stemming improves the results, as the experiment with no stemming scored below every other stemming approach. Despite HunSpell stemming scoring higher than no stemming, it should not be considered a bene cial approach, because the results do not correspond to the processing time that must be dedicated to the stemming. However, there are languages without any other stemming algorithms, which could bene t from the HunSpell stemming. This should be investigated further, especially in matters of processing time for these languages.

Another bene t of our participation was the addition of the Apache Solr interface. Now the Xtrieval framework is able to use Solr for indexing and retrieving documents. Furthermore existing retrieval interfaces like AJAX Solr3 can be used to inspect the index and the documents in a more interactive way than before.

References

[1] Kursten, J., Wilhelm , T. : Extensible retrieval and evaluation framework: Xtrieval . In Baumeister, J., Atzmuller, M., eds. : LWA. Volume 448 of Technical Report ., Department of Computer Science, University of Wurzburg, Germany ( 2008 ) 107 { 110

[2] Smiley , D. , Pugh , E.: Apache solr 3 enterprise search server ( 2011 )

[3] Kursten, J., Eibl , M. : Monolingual retrieval experiments with a domain-speci c document corpus at the chemnitz university of technology. [ 6 ] 178 { 185

[4] Halacsy , P. , Tron , V. : Bene ts of resource-based stemming in hungarian information retrieval . [ 6 ] 99 { 106

[5] Kursten, J., Wilhelm , T. , Richter , D. , Eibl , M. : Chemnitz at the chic evaluation lab 2012: Creating an xtrieval module for semantic enrichment . In Forner, P., Karlgren , J. , Womser-Hacker , C., eds.: CLEF (Online Working Notes/Labs/Workshop). ( 2012 )

[6] Peters , C. , Clough , P. , Gey , F.C. , Karlgren , J. , Magnini , B. , Oard , D.W., de Rijke, M. , Stempfhuber , M., eds.: Evaluation of Multilingual and Multi-modal Information Retrieval, 7th Workshop of the Cross-Language Evaluation Forum , CLEF 2006 , Alicante, Spain, September 20-22 , 2006 , Revised Selected Papers . In Peters, C. , Clough , P. , Gey , F.C. , Karlgren , J. , Magnini , B. , Oard , D.W., de Rijke, M. , Stempfhuber , M., eds. : CLEF . Volume 4730 of Lecture Notes in Computer Science., Springer ( 2007 )