In 2012, the University of Hildesheim participated in the CLEF-IP claims-to-passage task. 4 runs were submitted and di erent approaches tested. The tested approaches included a language independent trigram search approach, one approach formulating a query in the source language only and another approach with querys translated to English, German, French and Spanish. The results were not satisfactory and the task of passage retrieval as de ned in CLEF-IP proved to be di cult for current technology.
The University of Hildesheim integrated di erent approaches which included a language independent trigram search, one approach formulating a query in the topics source language only and another approach with querys translated to English, German, French and Spanish.
This years claims-to-passage task is based on manual relevance assessments from patent specialists taken out of search reports. This provides the participants with a realistic data basis, which should be positively emphasized. 2
In order to get a better understanding of the task and the relevance assessments, the topics and the qrels from the trainings set were examined from di erent perspectives. The following paragraphs describe our approaches and results. 2.1
Since the task is to nd relevant passages on the basis of given claims, it seemed natural to try to look for patterns and connections in the supplied training data. Given that the 51 topics were built from only 29 documents and relevant passages were often similar for two topics taken from the same document, we chose to examine ten topics (from distinct documents), which corresponds to about a third of the training data.
This rst manual analysis, which was complemented by using an Online Text Analysis Tool2, demonstrated that the absolute overlap of meaningful vocabulary (not counting stopwords nor regular patent terms) is minimal. In the ten documents, there were between two and seven matching terms in their base form comparing the query and the result set. In order to get to these comparable base forms, several methods like stemming, compound splitting and translation would have to be used. In three topics, two matching phrases consisting of two words (e.g. meat product) were also found. There was never a single overlap of longer phrases.
This analysis shows that it is extremely hard to nd relevant passages just on the basis of linguistic methods/matching terms. This knowledge led us to examining other approaches, e.g. the overlap of classi cation codes. 2.2
In real patent search scenarios, patent experts make immense use of the classi cation codes. Therefore, we analyzed the overlap of the classi cation codes from the topic documents and the documents with relevant passages. We automatically calculated the overlap of the IPCR-codes on the section, class and subclass level to be able to better decide if and where a cuto could be made during retrieval. 2 http://www.online-utility.org/text/analyzer.jsp
For the calcuation of the overlap we calculated how many of the relevant passages were assigned to at least one of the topic document's IPCR codes section, section & class or section & class & subclass. The results (Table 1) showed that using this constraint a very high recall could be achieved. It also showed that the high recall is accomplished only at the expense of precision and therefore the usage of IPCR codes alone will not su ce to reach satisfactory results. To get a better understanding on how to cope with multilingual challenges, the distribution of the languages in the data collection was analyzed. For the identi cation of the language we relied on the metadata supplied in the patent documents XML structure. The results were very interesting since they exposed that 40% of the documents neither contain abstract, nor claims nor description. The language distribution in the corpus can be seen in Table 2. The system used for both the training and the tests was based on Java and Apache Lucene and mostly on Lucene's built in language processing modules for language depending stemming, stopword removal etc. All translations were created with the help of the Microsoft Translator API. 3.1
Since the claim to passage task requires the matching of claims to passages in a patent, the passages in the patents were considered as documents themselves, and hence a passage based index was created. For each passage, the passage content was indexed and the patent document it appeared in was referenced by the patent document's le name.
For the training phase two indexes were created: a language independent trigram index (trigram index) and an index containing the passages original content as well as translations in clear text form (text index). Including the translations, the text index contained the passage content in the 4 most common languages in the CLEF 2012 corpus: English, German, French and Spanish.
For both the text index and the trigram index the passage content was processed before the actual indexing. The processing consisted of tokenization, lowercasing, stopword removal and stemming with language speci c tools provided by the Lucene project. Hereby the language was detected by the data supplied by in the patent document les' XML structure.
The indexes had the following elds in common: { le name: the name of the le the passage appeared in { IPCR all the IPCRcodes assigned to the patent document the passage appeared in { language: the language the passage was originally written in (according to the XML annotations in the patent document le) { type: the type of the passage, i.e. claim, description, heading or abstract
{ content de: German content { content en: English content { content fr: French content { content es: Spanish content
Instead of the language speci c content elds, the trigram index contains only one language independent content eld:
{ content: the processed content of the passage split into trigrams 3.2
Before searching, the contents of the topic passages were merged and subsequently processed. The text that was hereby created will from now on be referenced as topic text. The topic text was processed in the same way as the passage contents as described in Chap. 3.1.
Furthermore, extended stop word lists were created to reduce noise in queries. Since the topics are regular claims in the corpus, the 100 most common words in the corpus were manually analysed and searched for irrelevant words to nd a broader list of stop words. The extended lists were created for all the topic languages, i.e. English, German and French. The indexes remained untouched by the extended stop word lists.
3 di erent types of search were implemented:
After processing the text was split into trigrams. The search itself was implemented language independent and there was no distinction between trigrams of di erent languages.
The topic text was not translated and it was only used in its original language. A boolean query was formed to query every content eld in the clear text. A systematical outline of such a query: (content-de:TERM-1 OR content-en:TERM-1 OR content-fr:TERM-1) OR (content-de:TERM-2 OR content-en:TERM-2...) ... OR (content-de:TERM-X OR content-en:TERM-X...)
The topic text was translated using the Microsoft Translator. Boolean queries were formed, querying the language speci c eld with the respective translation of the topic text only. A systematical outline of such a query: (content-de:DE-TERM-1 OR content-de:DE-TERM-2 OR
content-de:DE-TERM-3 ... content-de:DE-TERM-X) OR (content-en:EN-TERM-1 OR content-en:EN-TERM-2 OR
content-en:EN-TERM-3 ... OR content-en:EN-TERM-X) ...
Additionally to those types of search, a set of parameters meant to change and improve the search results, were de ned. Those parameters were: { Boost Factor:
The Boost Factor, which is integrated into Lucene, was used to give a higher weight for reoccuring terms in the claims for search. For the calculation of the factor each term was boosted by the following formula was used: 1 + (BoostF actor (T ermF requency 1)) (1) { Passage Limit:
The maximum amount of total passages to be retrieved { Extended Stopwords:
Denotes whether the extended stopwords should be used or not. { IPCRLevel:
Should the IPCRCodes be considered, i.e. should paragraphs only be returned, if at least one of the IPCRCodes of the document they were found in is consistent to at least one of the IPCRCodes of the topic document. If the IPCRCode was to be used, the IPCRlevel, i.e. the number of characters of the IPCRCode to be considered, could be de ned as well. 3.3
As a baseline the search with single language topic text (S2) was chosen. As parameters, a Boost Factor of 0.5, a passage limit of 100, an IPCRlevel of 1 and no extended stopwords were used. With those settings, a precision of 0, a recall of 0.01 and an f-measure of 0 was achieved.
In order to improve those poor results, a batch tool was written, to allow the testing of several parameter con gurations and combinations at once. Using the training set, precision, recall and f-measure were calculated for those combinations. Using the batch tool, a wide set of parameters was tested but no combination resulted in satisfying results. The best results with a precision of 0.01, a recall of 0.02 and an f-measure of 0.02 were achieved with a boost factor of 0, a passage limit of 200, an IPCRlevel of 6 and without extended stop words, using the single language approach (S2). Those results were not signi cantly better than many other combinations resulting in a precision of 0, a recall of 0.1 and an f-measure of 0.1. 3.4
After the experiments, the con gurations that obtained the best results were chosen and handed in for the runs. This also meant that the trigram based approach was dropped and not handed in since it never yielded better results than the baseline. An overview of the runs and a description of their con gurations can be seen in Table 3. 3.5
4
The results (Table 4) show that the task of passage retrieval as de ned in CLEFIP is inherently di cult for current technology. The overall values are not satisfying. Only one group could achieve better results for the passage retrieval evaluation measures.
We observed that the terminology between the legal parts (claims) and the
technical parts (abstract and description) di ers quite substantially. In the future.
we intend to exploit these di erences and knowledge about the distribution of
terms in both parts. Optimized queries for the technical and the legal part of
the patent will be created and sent to the proper index elds. In addition. a
linguistic phrase parser [