For UFRGS's first participation on CLEF our goal was to compare the performance of heavier and lighter stemming strategies using the Portuguese data collections for Monolingual Ad-hoc retrieval. The results show that the safest strategy was to use the lighter alternative (reducing plural forms only). On a query-by-query analysis, full stemming achieved the highest improvement but also the biggest decrease in performance when compared to no stemming. In addition, statistical tests showed that the only significant improvement both in terms of mean average precision and precision at ten was achieved by our lighter stemmer.
This paper reports on monolingual information retrieval experiments that we have performed for CLEF2006. We took part on the ad-hoc monolingual track, focusing on the Portuguese test collections. Our aim was to compare the performance of lighter and heavier stemming alternatives. We compared two different algorithms: a Portuguese version of the Porter stemmer1 and the “Removedor de Sufixos da
Yes
No
Yes
No
No
Yes
Yes No
Each step has a set of rules, the rules in the steps are examined in sequence and only one rule in a step can apply. The longest possible suffix is always removed first because of the ordering of the rules within a step, e.g. the plural suffix –es should be tested before the suffix –s. At the moment, the Portuguese
• • • •
short. This measure varies for each suffix, and the values were set by observing lists of words ending in the given suffix. Although there is no linguistic support for this procedure it reduces overstemming errors. Overstemming is the removal of a sequence of characters that is part of the stem and not a suffix.
"inho", 3, ””, {"caminho", "carinho", "golfinho", "padrinho", "sobrinho", "vizinho"} "cominho", Where “inho” is a suffix that denotes diminutive, 3 is the minimum size for the stem, which prevents words like “linho” (linen) from being stemmed and the words between brackets are the exceptions for this rule, that is, they end in the suffix but they are not diminutives. All other words that end in –inho and that are longer than 6 characters will be stemmed. There is no replacement suffix in this rule.
Step 1: Plural Reduction With rare exceptions, the plural forms in Portuguese end in –s. However, not all words ending in –s denote plural, e.g. lápis, (pencil). This step consists basically in removing the final “s” of the words that are not listed as exceptions. Yet sometimes a few extra modifications are needed e.g. words ending in –ns should have that suffix replaced by “m” like in bons → bom.
Step 2: Feminine Reduction All nouns and adjectives in Portuguese have a gender. This step consists in transforming feminine forms to their corresponding masculine. Only words ending in –a are tested in this step but not all of them are converted, just the ones ending in the most common suffixes, e.g. chinesa → chinês.
Step 3: Adverb Reduction This is the shortest step of all, as there is just one suffix that denotes adverbs –mente. Again not all words with that ending are adverbs so an exception list is needed.
Step 4: Augmentative/Diminutive Reduction Portuguese nouns and adjectives present far more variant forms than their English counterparts. Words have augmentative, diminutive and superlative forms e.g. “small house” = casinha, where –inha is the suffix that indicates a diminutive. Those cases are treated by this step. According to (Cunha & LindleyCintra, 1985) there are 38 of these suffixes, however some of them are obsolete therefore, in order to avoid overstemming, our algorithm uses only the most common ones that are still in common usage. Step 5: Noun Suffix Reduction This step tests words against 61 noun (and adjective) endings. If a suffix is removed here, steps 6 and 7 are not executed.
Step 6: Verb Suffix Reduction
Portuguese is a very rich language in terms of verbal forms, while the regular verbs in English have just 4
variations (e.g. talk, talks, talked, talking), the Portuguese regular verbs have over 50 different forms
The Portuguese version of the Porter Stemmer and the RSLP are based solely on rules that need to be applied in a certain order. However there are some differences between the two stemmers: 2 There are 3 classes of verbs in Portuguese according to the ending of their infinitive form: “ar”, “er”, “ir”. Thematic Vowel the letter (“-a”, “-e” and “-i”) that groups verbs into categories. • • •
The number of rules – RSLP has many more rules than the Portuguese Porter because it was designed specifically for Portuguese. There are some morphological changes such as augmentatives and feminine forms that are not treated by the Portuguese Porter.
The use of exceptions lists – RSLP includes a list of exceptions for each rule as they help reducing overstemming errors.
3
This section describes our experiments submitted to the CLEF-2006 campaign. Section 3.1 details the resources used, and Section 3.2 presents the results. 3.1
Description of Runs and Resources The Portuguese data collections were indexed using SMART3. We used the title and description fields of the query topics. Query terms were automatically extracted from the topics. Stop words were removed from both documents and topics. In addition, terms such as “find documents” were removed from the topics. The processing time was less than 4 minutes for all runs. This includes indexing the 210,734 documents and running all 50 queries.
• • • •
3.2
Results
3 Available from ftp://ftp.cs.cornell.edu/pub/smart/
80,00%
70,00%
60,00%
50,00%
n
o
i
is40,00%
c
e
r
P30,00%
20,00%
10,00%
0,00%
RSLP
Porter
RSLP-s
No Stemming
In order to tell whether the performance improvements shown in Table 1 are statistically significant, a
paired T-test was performed. Although our data is not perfectly normally distributed, Hull
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Recall A query-by-query, analysis shown in Table 3, demonstrates that for 12 topics no stemming was the best alternative. Some form of stemming helped 38 out of 50 topics. Confirming the results in terms of MAP and Pr@10, the best performance was achieved by the lighter stemming alternative RSLP-S. Full stemming with RSLP achieved the biggest performance improvement (topic 340 AvP 0.0003 → 0.3039), but also the biggest drop (topic 343 AVP 0.4276 → 0.1243). Stemming also helped finding 221 relevant documents that were not retrieved by the NoStem run. It seemed plausible that queries with few relevant documents would benefit more from stemming, resulting in a negative correlation between the number of relevant documents for the topic and the change in performance achieved with stemming. However a weak positive correlation of 0.15 was found. We would like to be able to predict the types of queries that would be benefited from stemming, but that needs further analysis with a larger number of topics. This paper reported on monolingual ad-hoc IR experiments using Portuguese test collections. We evaluated the validity of stemming comparing the Portuguese version of the Porter stemmer and two versions of the RSLP stemmer, one that applies full stemming and one that only reduces plural forms
• • •
The lighter version of the RSLP stemmer yields statistically significant performance improvements both in terms of MAP and Pr@10.
Full stemming, both with Porter and RSLP, has improved the results in terms of MAP and
This work was supported by a CAPES-PRODOC grant from the Brazilian Ministry of Education. Cunha, C., Lindley-Cintra, L., 1985. Nova Gramática do Português Contemporâneo. Rio de