In the context of the CLEF-IP 2010 classi cation task, we conducted a series of experiments with the Linguistic Classi cation System (LCS). We compared two document representations for patent abstracts: a bag-of-words representation and a syntactic/semantic representation containing both words and dependency triples. We evaluated two types of output: using a xed cut-o on the ranking of the classes and using a exible cut-o based on a threshold on the classi cation scores. Using the Winnow classi er, we obtained an improvement in classi cation scores when triples are added to the bag of words. However, our results are remarkably better on a held-out subset of the target data than on the 2 000-topic test set. The main ndings of this paper are: (1) adding dependency triples to words has a positive e ect on classi cation accuracy and (2) selecting classes by using a threshold on the classi cation scores instead of returning a xed number of classes per document improves classi cation scores while at the same time it lowers the number of classes needs to be judged manually by the professionals at the patent o ce.
The data selection in our experiments was motivated by practical concerns: Since we wanted a comparison between classi cation experiments using bag-of-words and syntactic/semantic terms, the choice of data was limited to abstracts as these are the easiest and consequently the fastest to parse. We parsed all (over 500,000) English abstracts of the corpus in a couple of days. To parse all the claims and/or description sections would have taken considerably longer because of the extremely long and complex sentences used in these sections. [4].
We extracted from the corpus all les that contain both an abstract in English and at least one IPC class in the eld <classification-ipcr>.2 We extracted the IPC classes on the document level, not the invention level. This means that we did not include the inventions where the IPC class is in another le than the English abstract. We saved the abstract texts in plain text, and administrated the IPC classes in a separate le.
For the bag-of-words representation, we ran a simple normalization script that removed punctuation, capitalization and numbers from all abstract les. For the syntactic/semantic representation, we parsed the abstract texts with the AEGIR dependency parser [3]. AEGIR allows us to set a maximum parse time per sentence, which is useful since for longer (and hence more ambiguous) sentences the parsing speed goes down. The output of the parser is a list of dependency triples for each abstract that have undergone a number of normalizing transformations on the morphologic and syntactic level, such as the transformation from passive to active voice (hence the term `syntactic/semantic').
Figure 1 is an example of a small original text, the normalized text in the bag-of-words representation and the triples in the syntactic/semantic representation. For the experiments with the syntactic/semantic representation, the triples are concatenated to the words. Table 1 gives general statistics on the target data and the test data.
Original text words triples Heat is stored heat is stored [IT,SUBJ,store] [store,OBJ,heat] at a steady at a steady [store,PREPat,temperature] temperature using temperature using [temperature,ATTR,steady] calcium chloride calcium chloride [temperature,DET,a] [chloride,ATTR,calcium] hexahydrate and hexahydrate and [hexahydrate,ATTR,chloride] up to 20 percent up to percent [hexahydrate,ATTR,using] [up,PREPto,20 percent] strontium chloride strontium chloride [assist,OBJ,crystallization] hexahydrate hexahydrate [chloride,ATTR,strontium] to assist to assist [hexahydrate,ATTR,chloride] crystallisation. crystallisation [hexahydrate,SUBJ,assist] 2 IPC-R is the IPC Reform classi cation, sometimes http://www.intellogist.com/wiki/IPC Classi cation System.
For our classi cation experiments, we use the Linguistic Classi cation System (LCS)3 [2, 1]. The LCS can perform both mono-classi cation (each document belongs to precisely one class) and multi-classi cation. In the training phase, the LCS takes as input a le which list the paths to the classi cation les followed by their classes. After this training phase the LCS can be used for testing the classi er obtained on a test collection of documents with known classes (usually held-out training data), or for producing a classi cation of new documents without known classes. 3.1
Three classi ers have been implemented in the LCS: Naive Bayes, Winnow or SV M light. We experimented with both Winnow and SV M light and we found that their classi cation accuracy scores are comparable but that SV M light is much slower. For example, in order to train a model based on 425 819 abstracts that belong to 629 di erent subclasses, Winnow needed around two hours (independent of the document representation used) while SV M light spent six and a half hours on the same task. Therefore, we decided to use Winnow for the CLEF-IP experiments.
Winnow has a number of parameters that can be tuned: , and maxiters (the number of training iterations). After some tuning around the default values, we decided to use = 1:02 and = 0:98. For maxiters, we experimented with three and ten iterations, and found that the classi cation accuracy still improved somewhat after the third iteration. Therefore we decided to use ten iterations
In the case of multi-classi cation, LCS is exible with respect to the number of classes that is
returned per document. Internally, it produces a full ranking of classes for each document in the
test set. The user can regulate the selection of classes with three parameters: (
We present the results on four experiments with LCS: 1. Classifying abstracts from the target data in the bag-of-words (words-only) representation into
IPC-R subclasses 2. Classifying abstracts from the target data in the syntactic/semantic (words+triples) representation into IPC-R subclasses 3. Classifying abstracts from the test data in the bag-of-words (words-only) representation into
IPC-R subclasses 4. Classifying abstracts from the test data in the syntactic/semantic (words+triples) representation into IPC-R subclasses For experiments 1 and 2, we randomly split the target data: we used 80% of the data for training the classi er and 20% for testing. We repeated this four times with di erent random splits and calculated the mean and standard deviation over the four outcomes in order to get a measure for the reliability of the results. For experiments 3 and 4, we applied classi cation models which were previously trained on a random 80% of the target data to the 2 000 abstracts from the test data, after the relevance assessments for the topics had been released by the organization. 3.2
We present the results in terms of precision (P ), recall (R) and their harmonic mean (F1) for two types of output: (a) the classes that were selected using the threshold on classi cation scores in 3 A demo of the application can be found at http://ir-facility.net/news/linguistic-classi cation-systemprototype/ for registered IRF members.
LCS and (b) the classes that were returned using a xed cut-o point in the class ranking. For the threshold-based cut-o , precision and recall are calculated using: For the xed cut-o , precision and recall are calculated using:
P = jrelevant classes \ selected classesj
jselected classesj R = jrelevant classes \ selected classesj
jrelevant classesj
R@n = jrelevant classes \ classes returned@nj
jrelevant classesj
(
With respect to the rst point, we observe a signi cant improvement in classi cation performance on the target data when we add triples to the bag of words: F1 increases from 64.30 (with a standard deviation of 0.08) to 67.16 (with a standard deviation of 0.07). However, on the test data, this di erence is much smaller and probably not signi cant.4 4 We cannot measure standard deviations for the test data because the topic set is too small to split up and compare the results on random subsets of it.
That brings us to the second point: the results for the target data and test data are very di erent from each other. Overall classi cation scores are lower for the topic test set than they are on a held-out set from the target data (F1 for words-only is 55.77 compared to 64.30). Inspection of the les in both sets shows that all les included in the test data are newer than the ones in the target data. This was done by the CLEF-IP organization to re ect the realistic task of classifying incoming patent applications using a model trained on existing patents. The fact that models trained on older abstracts are a better t on contemporary abstracts than on more recent abstracts suggests that the content of the patents belonging to a speci c subclass has changed over time.
It is more di cult to explain why the improvement gained from adding triples to words is smaller for the test data than it is for the target data. Table 1 shows that in the test data more abstracts had empty parser output than in the target data but this di erence is small (4.6% and 3.5% respectively). We checked the output of the parser for the topic abstracts but we have no reason to believe that the topic abstracts were that much more di cult to parse as to result in less reliable triplets. This leaves us with the option that the smaller improvement is (at least partly) due to coincidence. There are only 2 000 topic abstracts that are classi ed in 476 di erent IPC-R classes. A di erent selection of 2 000 abstracts could easily lead to a few percent change in the classi cation accuracy.
Finally, we compared the results on the ranking with xed cut-o to the results for the threshold-based cut-o . We see that class selection using a threshold on the classi cation score has a positive e ect on both the precision and the recall, and hence on the F1 score (64.30% compared to 57.01% at rank 4 for words-only on the target data). Selecting classes by using a threshold on the classi cation scores for the classes instead of returning a xed number of classes per document leads to better classi cation while a lower number of classes needs to be judged manually. 5
For the proceedings of CLEF-IP 2010, we plan to conduct follow-up experiments in two directions.
First, we will investigate why the improvement gained from adding triples to words is smaller
for the test data than it is for the target data. We plan to look into (
In order to nd out whether the di erences between the results for the test data and the target data are due to coincidence, we plan to create at least ve test sets of 2 000 abstracts extracted from the same time slice of the MAREC corpus as the supplied topic test set. Then we will classify these sets using the same models trained on the target data in order to obtain the variation of the classi cation accuracy on test sets of 2 000 abstracts.
Fine, we plan to set up a series of tuning experiments for the threshold parameter in LCS on a held-out development set, to see if we can gain additional improvement from optimizing the class selection.