For the 2012 CLEF-IP Claims to passage task we reused and improved our Xtrieval framework. Our two-step approach comprises creating two Lucene indexes: one containing the whole patent application documents and one containing the same documents split into passages. We prepared three setups and conducted each with a translated and an untranslated topic set, which was just applied to the claims. The submitted setups di er in the way of retrieving the results and merging them. No further techniques were used. Therefore our experiments had very simple setups, which nevertheless achieved good results. There are still plenty of possible improvements, which can easily be tested with our framework, because it o ers a comprehensive set of methods for conducting and evaluating retrieval experiments.
In the course of our participation we re-engineered the Xtrieval framework. We further consolidated it and enhanced it for more speed. The speed improvements were necessary because of the size of the data collection (26 gigabytes 1 MAREC IRF, http://www.ir-facility.org/prototypes/marec 2 Information Retrieval Facility, http://www.ir-facility.org/ 3 Apache Lucene (http://lucene.apache.org/) and Terrier (http://terrier.org/) compressed, 107 gigabytes uncompressed) and the necessity of executing several iteration of the experiments.
Our experiments last year showed that longer queries outperform shorter
ones[
We adapted this approach to the claims to passage scenario by creating two indexes: one containing the whole patent application documents and one containing the same documents split into passages.
For the passage index the following XPaths were use: /patent-document/description/p /patent-document/description/heading /patent-document/claims/claim /patent-document/abstract/p
/patent-document/abstract/p /patent-document/description/* /patent-document/claims/claim /patent-document/bibliographic-data/technical-data/invention-title
The Xtrieval framework has a very exible and fast implementation for reading XML data collections based on the Jaxen library4. It exclusively relies on XPath for selecting the content and determining the destination elds in the index.
For the retrieval phase we prepared three di erent setups and conducted each with a translated and an untranslated topic set. We did not translate the whole patent but just the claims referenced in the topics. For the translation we used Google's Translator Toolkit5 and translated all claims to the three most used languages of the data collection: English, German, and French. While other languages occur in the corpus, these three languages provide the vast majority of content according to an intermediate index we created.
All three setups share the following pre-processing steps: 1. instead of relying on the given language attributes we used a language detection6 to eliminate content tagged wrong 2. bag of words 4 jaxen: universal Java XPath engine, http://jaxen.codehaus.org 5 Google Translator Toolkit, http://translate.google.com/toolkit/ 6 http://code.google.com/p/language-detection/
We used the standard tokenizer of Lucene, which splits a text stream into tokens and recognizes some entities like URLs and e-mail addresses. Then we applied our own lters (marked with *) and some from the Lucene package. If the lter depends on the token language this was considered for the following languages: English, German, French, Russian, Italian, and Spanish.
2. RemoveShortWordsFilter * - removes words shorter than 3 characters 3. StopFilter - removes stop words depending on the language 4. RemoveNumbersFilter * - removes di erent kinds of numbers 5. SnowballFilter 7 - stems the token according to its language
The following setups di er just in the way of retrieving the results and merging them. No further techniques were used. That is to say: no elds (bag of words) or eld weights, no relevance feedback, no language model, and no further query expansion. 2.1
This setup should be considered as our baseline, because we only used the claims speci ed in the topics and searched them in the passage index. 2.2
In this setup the query is constructed by merging the patent documents and the extracted claims. The content of the patent documents got a lesser weight than the claims to focus more on the claims. The queries were issued just on the passage index. 2.3
Our most sophisticated setup was the two-step approach, which was mentioned earlier. In the rst step we retrieved a set of potentially relevant patent documents. For the second step their identi ers were used to amend the query to the passage index. The identi ers in the passage query are just optional to not exclude passages, which are still relevant but are not included in the rst step. Some experiments with the provided test set showed beforehand, that limiting the second step to the results obtained in the rst one will achieve a signi cant lower score. 7 Snowball, http://snowball.tartarus.org/
Our focus this year was especially on improving and testing the framework. Therefore our experiments had very simple setups, which nevertheless achieved good results. The passages-only run without translation should be rated as our baseline and all other runs compared with it. This comparison shows that the only run outperforming our baseline, except for the precision at passage level, is our two-step approach without any translation (tuc-d2-p). All other runs achieved lower scores, which show they are not suited to improve the retrieval.
Because of the improved speed for the index and retrieval process in our framework, we could iterate more and experiment with di erent weights for the di erent combinations of documents and passages.
There are still plenty of possible improvements, which can easily be tested
with our framework: pre-tokenization lters, token lters, retrieval systems (i.e.
Lucene and Terrier), query expansion, and query reformulation. As the
framework supports the calculation of di erent measures (i.e. PRES@n, MAP, and
so on) one can compare the results with previous experiments. This could more
easily be done with tools like EvaluatIR[