With the emergence of vast resources of information, it is necessary to develop methods that retrieve most relevant information according to the users needs. These retrieval methods may benefit from natural language constructs to boost their results by achieving higher precision/recall rates. In this attempt, we have used part of speech attributes of terms as extra information about document and query terms and have evaluated the impact of such information on the performance of the retrieval algorithms. Also the effect of stemming has been experimented as a complement to this research. Our findings indicate that part of speech tags may have small influence on effectiveness of the retrieved results. However, when this information is combined with stemming it improves the accuracy of the outcomes considerably.
l a v e i r t e R n o i t a m r o f n I n a i s r e P n I g n i g g a T h c e e p S f o t r a P g n i s
U 1 . I n t r o d u c t i o n Exploiting meta-information of the terms in the retrieval process can result in precision and recall improvements. Part of Speech tagging techniques can be used for this purpose by clarifying the role of the terms in the queries and documents and consequently making it possible to assign different priorities to different query terms based on their part of speech. In addition stemming can collapse many surface words in languages like Arabic and Persian into a single representation and improve the recall of the system.
The general objective of the present study is to further investigate the potential benefits of incorporating
part of speech information into both query statements and the document collection in the Persian
language and to observe the consequences of such incorporation in document retrieval. This general
objective is also complemented by the investigating the effect of stemming in such environment.
2 . L i t e r a t u r e R e v i e w
Different algorithms and methods have been developed to make more accurate retrieval engines
There has been much work on Persian Information Retrieval but none of these approaches have used part
of speech tagging which has been tested on other language leading to good results
In this study, we used Bijankhan
U
It has been reported that in some applications of IR, the nouns are more important than other tokens
After analyzing the impact of these 40 different tags, eventually we find out that noun, verb, adjective and adverb are the most important POS Tags in Persian retrieval. In the result section we show the impact of using these tags on documents in the corpus and also queries.
The TnT POS tagger8 is used in this study to determine the part of speech of Persian words
After experimenting with different tagging schemas, the corpus and the queries are then stemmed to evaluate the consequences of stemming on the retrieval system. In addition the tagged corpus will be stemmed too in order to find out how these two approaches may complement each other. 8 TnT can be found at http://www.coli.uni-saarland.de/~thorsten/tnt/ l a v e i r t e R n o i t a m r o f n I n a i s r e P n I g n i g g a T h c e e p S f o t r a P g n i s
U 5 . I m p l e m e n t a t i o n Using Parts of speech tagging metadata information as our main approach, we developed configurations from various combinations of term stemming beside POS tagging in order to observe its effectiveness in action. As stated in previous section tagged corpus was obtained using statistical techniques. Bijankhan tagged collection was used as the training data to estimate the role of each word in the Hamshahri collection. Stemming was performed by employing simple grammatical rules using PERSTEM Persian stemmer9. Consequently we prepared 4 different collections as shown in Table 2. Normal (Neither stemmed nor tagged)
Terms tagged with related parts of speech
9 http://sourceforge.net/projects/perstem
TABLE 3 DIFFERENT CONFIGURATIONS Config. Corpus
Query Title with equal weighting for all POS tags Stemmed title with equal weighting for all POS tags Stemmed title without POS tagging Stemmed Title plus description Title plus description with equal weighting for all POS tags Title with various weighting schemes for different POS tags
Title (Neither stemmed nor tagged) l a v e i r t e R n o i t a m r o f n I n a i s r e P n I g n i g g a T h c e e p S f o t r a P g n i s
U 6 . R e s u l t s Before going through the results, it’s worth mentioning that since the Hamshahri collection has never been tagged before, we do not have any measurement of the accuracy of the POS tags however basic observations and sampling shows reasonable accuracy. 10 The Lemur Project. 2001-2008. University of Massachusetts and Carnegie Mellon University. [www.lemurproject.org] l a v e i r t e R n o i t a m r o f n I n a i s r e P n I g n i g g a T h c e e p S f o t r a P g n i s
U
TABLE 4 MAIN RESULTS normal corpus tagged (title) tagged (title
and description)
The results we obtained indicate that the Persian retrieval benefits from stemming. Stemming the documents and queries alone returned one of the best results of our experiments with the average precision at 36% and R-Precision at 41%. This is in contrast with experiments conducted by other groups in University of Tehran on the same corpus. Therefore further experiments with different types of stemmers and stemming techniques are required in order to clarify the role of stemming in Persian text processing.
Even though stemming increases the performance of the system, there is a sudden fall when the description is added to the queries, this time even worse than our base line system. The reason for this again goes back to the expansion of the queries by words which might be unnecessary for our results, but this time since many words have turned into a single representation, the effect of misleading terms have increased dramatically. This undesired performance does not get any better even with the omission of stop words that seem to play an important role.
The best result of our experiment was achieved by using the stemming schema on the tagged corpus and title of the queries. This time the average precision reached its peak at 39% and the R-precision stayed at 41% like the stemmed corpus without tags which could mean that the effect of stemming on our retrieval system is much higher than part of speech tagging.
The main focus of our experiments was the weighting of the different terms based on their tags. Many different combinations of weights for different tags have been tried but not only had none of the weighing schemes improved the performance, some of them demonstrated the worse performances. Table 5 depicts the results of some of these experiments. Giving a weight of zero to a tag is the same as omitting that tag from the corpus and the queries. In some experiments as much as 20 least significant tags were omitted from the queries but the precision and recall were reduced. In general the average precision for all the tag weighting schemas was 22% and the average R-Precision was 26% as seen in table 4.
TABLE 5 WEIGHTING SCHEMAS
20 less used tags omitted, others equal weight The reason for such behavior can be explained by the importance of different tags in the Persian language. Despite our original study that led us to the omission of the 20 least important tags, they actually played a role in the retrieval. Thus by omitting them the performance of the system declined. l a v e i r t e R n o i t a m r o f n I n a i s r e P n I g n i g g a T h c e e p S f o t r a P g n i s
U
FIGURE 2 R-PRECISION OF THE DIFFERENT CONFIGURATIONS 7 . C L E F 2 0 0 8 R e s u l t s All the preceding results were based on training data that is a part of Hamshahri Corpus. In order to further benchmark our system we decided to participate in CELF 2008 monolingual Persian ad hoc track. Our system was named Tehran-NLP and its results were not promising and ranked very low among the other participants. There are some explanations as: • •
Our model was developed on the basis of including meta-information about the part of speech of words in information retrieval and since this was our first participation in CLEF 2008, our system was not fine tuned for CLEF.
Not knowing the rules correctly, we submitted the top 100 retrieved documents instead of top 1000 as all other University of Tehran participants did. This could have an undesirable effect on our runs.
As a result our system was not that successful in CLEF 2008 and we could not make it to the top five. 8 . C o n c l u s i o n a n d F u t u r e W o r k This study attempted to measure the effectiveness of part of speech tagging and stemming on Persian information retrieval. Different configurations were used as weighting schemas and the first set of results on training data showed that tagging does improve the performance of the system while weighting based on POS tags reduces it.
The second part of our studies demonstrated that stemming when used along with tagging has a significant effect on the performance. l a v e i r t e R n o i t a m r o f n I n a i s r e P n I g n i g g a T h c e e p S f o t r a P g n i s
U
The result of this study has several implications for future research, first all POS tags are important. Omitting words with specific parts of speech tags in queries or even in documents can have negative effect on Precision and Recall.
We used the simplified tag set with only 40 tags and simplified version of Bijankhan Tagged corpus as our training data. After training we tagged the entire Hamshahri collection automatically. For future research, one might consider a more complex set of part of speech tags. For example, it seems reasonable to assume distinguishing location names from general nouns could be important for query processing and might have a positive effect on effectiveness.
In this research a simple grammatical based algorithm was used to stem Hamshahri collection. Other research groups in DBRG lab of University of Tehran have tried other stemming algorithms and methods and have come up with different results. So, in future, one might want to conduct experiments with all the available stemming methods and algorithms and come up with an explanation for these different behaviors. In addition, other retrieval models should be studied in these experimental configurations in order to ensure that the results obtained are general and not dependent on a particular retrieval model such as Indri.
R e f e r e n c e s