In 2011, the University of Hildesheim and Chemnitz University of Technology participated together in the CLEF Intellectual Property Track. We focused on the prior art candidate search, which was already provided for the third time. Our group submitted seven runs ranging from simple bag of words to linguistic phrases. The aim of our experiments was to examine the effectiveness of different query strategies. Especially, we wanted to evaluate the advantage of linguistic phrases in contrast to very long bag of words queries. Phrases were extracted using a special extraction component, which has been developed by the University of Hildesheim.
Prior art search is a well known type of patent search that is performed to find out whether there exists prior art to a given patent application [1, 2]. In contrast, the goal of the classification tasks is to classify patents according to the International Patent Classification (IPC). Furthermore, two image-based tasks were introduced this year [1].
At CLEF-IP 2011, the University of Hildesheim and Chemnitz University of Technology did joint work. Each of our experiments concentrated on the prior art search, which has been organized for three years now.
The test collection, which was provided by the IRF1, consisted of approximately 2.5 million documents stored as XML files. Each of the documents from the European Patent Office (EPO) was published before 2002 [1, 3]. The collection differed with respect to the last two years, because it contained about 400.000 extra documents from the World Intellectual Property Office (WIPO) [1, 3]. Beside the document collection, the organizers provided one topic set (about 4.000 patents assigned the code “A1” or “A2”), which equally consisted of German, English and French topic files [1].
Our experiments were performed on the basis of the Xtrieval framework developed at
Chemnitz University of Technology. This framework consists of four different
components, three of which form the system core (
The Xtrieval framework was designed to make use of common retrieval API's, such as Lemur2, Terrier3 and Apache Lucene4, for evaluation purposes. For the present experiments we used Apache Lucene in version 3.1 as retrieval core in Xtrieval [4]. Thus, the underlying retrieval model is the traditional Vector Space Model. More details on our approach are given in the following section. 1 Information Retrieval Facility 2 http://www.lemurproject.org/ 3 http://terrier.org/ 4 http://lucene.apache.org/
To index the document collection, the standard retrieval approach was followed. As a consequence, the text extracted from the XML documents was first preprocessed. This preprocessing included the following steps: • • • stopword elimination tokenization stemming In [5] it was mentioned that patent specific terms, which appear frequently are likely to result in comprehensive document lists. Following this, our group decided to use a customized stopword list, i.e. a standard stopword list5 which was amended with a number of patent specific terms. This approach was already used in 2009 and 2010 [5, 6]. In a next step, the preprocessed text was stored into the index file. Because it proofed to be more effective during the experiments on the trainings set, we stored the text as a bag of words. In this context, each language was treated separately. Thus, the resulting index consisted of three fields, one single field per language. Furthermore, the following patent parts were considered during the indexing process: • • • •
Besides this textual information, the language-independent IPC codes were included into the index, because the experiments at CLEF 2009 showed that these are particularly advantageous to increase the recall of an information retrieval system [5].
A lot of research has concentrated on the effectiveness of different kinds of phrases. At CLEF 2010, the University of Hildesheim investigated the effectiveness of terms and phrases in the context of patent information retrieval. The experiments using phrases significantly outperformed the term baseline, although a simple statistical approach was used [6]. Furthermore, in [7] the author focused on the advantage of statistical and syntactical noun phrases for interactive query expansion. In this case, linguistic phrases proofed to be effective for information retrieval [7]. Following this, we decided to investigate the influence of linguistic (dependency) phrases on the effectiveness of a patent retrieval system. In this context, different types of phrases were considered ranging from simple adjective noun to complex noun object relations.
To run our experiments we extracted phrases using a special extraction component that has been developed at the University of Hildesheim. The underlying approach is called rule based dependency parsing and combines the rule based method described in [8] with dependency parsing. In case of rule based dependency parsing, dependency phrases are identified with the help of defined term pairs. The following example illustrates this approach: A METHOD FOR IMPLEMENTING ONLINE MAINTENANCE COMMUNICATION NETWORK (Topic EP-1881641-A1) IN
THE In the example title the tool would extract the phrase “method for implementing online maintenance”. The determiner “a” and the preposition “in” serve as patterns to identify the phrase. This approach was implemented using UIMA6 and openNLP7 as the basis of the extraction tool. A detailed description of the system developed by the University of Hildesheim can be found in [9]. By now, the extraction tool has been tested only on English documents. As a consequence, the phrase experiments concentrated on English patents only.
Our group performed various experiments for the prior candidate search task. Given the topic file, the goal was to identify those patents that describe prior art. As already mentioned, a topic file is a patent which is assigned code “A1” or “A2” [1]. On the basis of these documents, the query was constructed automatically from the content of different patent parts. These included the following: • • • • •
Abstract Claim Description
IPC
At CLEF-IP 2011, we experimented with the following query modifications:
1. Boolean queries consisting of terms (
The University of Hildesheim and Chemnitz University of Technology submitted seven different runs. Each experiment made use of the same index file, but differed with respect to the query options. A detailed description of these runs as well as an overview of the results can be found in the next two sections.
Our group performed four multilingual (
1. 2. 3. 4. 5. 6. 7.
CUT_UHI_CLEFIP_BOW: query terms extracted from abstract, claims and title CUT_UHI_CLEFIP_BOW_DESC: query terms extracted from abstract, claims, title and description
abstract, claims, title and description, IPC CUT_UHI_CLEFIP_BOW_IPCR: query terms extracted from abstract, title and claims, IPC
from abstract and title CUT_UHI_CLEFIP_BOW_EN_P: linguistic phrases extracted from abstract and terms of title
extracted from abstract and terms extracted from abstract and title phrases As can be seen, only the third experiment made use of all patent sections and the other runs were restricted to special fields. For example, the first run was restricted to abstract, claims and title. We did not use the whole IPC code, but included the first four digits only. Although the complete classification information proofed to be more accurate, we decided to use only the first four digits, because this significantly accelerated the search process.
Our runs were divided into two major categories. The first group of experiments (
The results revealed that the first query strategy, which was based on using as many terms as possible, proofed to be more effective, because our best run achieved a map of 0.0914 (run 3). In this case, the query was formulated using terms extracted from all patent fields. Furthermore, the experiment that concentrated on claims, title and abstract (run 1) proofed to be effective (0.0824). Surprisingly, the recall of this run was similar to that of the fourth experiment (0.4318) using the IPC additionally. This indicates that the classification codes do not have any advantage over the title, abstract and claims. In contrast, the run that additionally concentrated on the description (run 2) achieved a lower recall (0.3993). Following this, we could summarize that the detailed description seems to be more advantageous with respect to the precision of a patent retrieval system while the remaining patent sections do have a positive effect on the completeness of the search results. Some statistics according to the obtained results are provided in Table 1. The results in Table 1 further indicate that using linguistic phrases did not increase the effectiveness of the retrieval system. Instead, the results of this experiment (0.1899; 0.0208) did significantly fall below the recall and map values of the remaining runs. This aspect is quite surprising, because phrases, in general, are considered to be more effective than terms [7]. One reason for the negative results of this experiment might be the small number of phrases that were extracted from the abstract. For example, only three linguistic phrases were extracted from topic EP-1226990 with the help of the extraction tool. The enrichment of phrases with additional terms (run 6) led to higher recall and map values. This could be a hint that our phrase queries were too short.
Furthermore, the abstract might not be the adequate patent section to construct phrase queries because of the existence of noisy terms [10]. As can be seen in the above table, integrating the terms from the detailed description (run 2 and 3) achieved the best results. Therefore, this patent section might be a better basis for generating phrase queries.
Besides the retrieval effectiveness, the duration of the experiment and the query length were measured. Some statistics of these aspects are displayed in the following table.
Although it seems to be quite effective, the use of very long queries has one disadvantage, because the search process is slowed down. As can be seen, the experiment which concentrated on the description (run 2) ran about eleven hours. In case of a realistic prior art search this might be problematic.
At CLEF 2011, the University of Hildesheim and Chemnitz University of Technology did joint work. Our experiments concentrated on the following two different aspects: 1. The effect of linguistic phrases extracted from patent documents 2. The effect of very long queries with maximum number of terms The results reveal that very long queries seem to be more effective in the context of patent information retrieval, but they significantly slow down the search process. In contrast, short queries which were constructed of phrases extracted from the abstract did not show any positive effect on neither recall nor mean average precision, but they are advantageous with respect to the run time. This raises the question of how long a patent searcher is willing to wait for accurate search results.
In the future, we will have to think about a combined search strategy that takes into account terms as well as phrases, because both query types seem to have some advantages. Furthermore, we will have to improve the generation of phrase queries, because phrases extracted from the abstract did not improve the retrieval effectiveness. Using a different part of the patent, e.g. the detailed description, might show some improvements with respect to recall or map.