Abstract

This paper describes the first participation of the SINAI team in the CLEF 2007 CLSR track. This year, we only want to establish a first contact with the task and the collections. Thus, we have pre-processed the collection using the Information Gain technique in order to filter the labels with most relevant information. We have used the LEMUR toolkit as the Information Retrieval system in our experiments.

Keywords

• Spoken Document Retrieval
• Information Gain
• Label filtering

-

Introduction

Label Selection with Information Gain We have used the Information Gain measure [ 7 ] to select the best XML tags in the collection. Once the document collection was generated, experiments were conducted with the LEMUR retrieval information system, applying the Kl-divergence weighing scheme.

The method applied consists in computing the Information Gain for each label in the collection. Let C be the set of cases and E the value set for the E tag. Then, the formula that we have to compute must obey the following expression:

IG(C|E) = H(C) − H(C|E) where

IG(C|E)

H(C) H(C|E) is the Information Gain for the E label, is the entropy and of the set of cases C is the relative entropy of the set of cases C conditioned by the E label

Both, H(C) and H(C|E) are calculated basing them on the frequencies of occurrence of the labels according to the combination of words which they represent. After some basic operations, the final equation for the computation of the Information Gain supplied by a given tag E over the set of cases C is defined as follows:

IG(C|E) = − log2 |C| + |E| |Cej | log2 |Cej | 1 1

j=1 |C|

For each tag in the collection, its Information Gain is computed. Then, the tags selected to compose the final collection are those showing higher values of Information Gain. Once the document collection was generated, experiments were conducted with the LEMUR retrieval information system, by applying the Kl-divergence weighing scheme. 3

Experiment Description and Results Our main goal is to study the effectiveness of filtering tags using Information Gain in the text collection. For that purpose, we have accomplished several experiments using all the tags in the collection to identify the best tag percentage with experiments preserving 10%, 20%...100% of tags (Figure 1). It is important to note that rare values of a label lead to very high Information Gain values as said for the DOCNO label, whose values are unique for each document. This is the expected behaviour for Information Gain, because by knowing the DOCNO label we could retrieve the exact document. Unfortunately, this label is useless, since we expect to retrieve documents based on the content of the other documents. For this reason we calculate a new value based on document frecuency (DF). The labels with low DF are put in the bottom of the list. Table 1 shows the Information Gain values of the collection labels, sorted by Information Gain and aplicating DF reordering.

Therefore, we have run ten experiments (with ten Information Gain collections) for each list of topics in English, Dutch, French, German and Spanish. However, we have only sent five runs, since the organization limited the number of submits.

French, German and Spanish topics have been translated to English using a translation module.

As translation module we have used SINTRAM (SINai TRAnslation Module), our Machine Translation system that works with different online machine translators and that implements some heuristics[ 8 ].

SINTRAM uses some online Machine Translators for each language pair and implements some heuristics to combine the different translations. After a complete research we have found that the best translators were: (1) (2)

Collection 10%

IG Filtering

Collection 20%

Collection 30%

...

List of labels sort by IG

Collection 100%

In our first participation in CLEF CL-SR we have used Information Gain in order to find the best tags in the collection. The experiment accomplished show the best tags is the SUMMARY label. However, the obtained results are not successful. At the moment we are investigating the reasons of these unexpected results.

Nevertheless, results of cross-lingual experiments show that Spanish and Dutch translation are better than other experiments. The Spanish experiments confirm the good results obtained in the ImageCLEF ad-hoc task this year.

Labels IG DOC/SUMMARY 12.9834 DOC/ASRTEXT2004A 12.9792 DOC/ASRTEXT2006B 12.9775 DOC/AUTOKEYWORD2004A2 12.9574 DOC/AUTOKEYWORD2004A1 12.9521 DOC/ASRTEXT2006A 12.6676 DOC/MANUALKEYWORD 12.6091 DOC/ASRTEXT2003A 12.5953 DOC/NAME 11.9277 DOC/INTERVIEWDATA 8.4755 DOC/DOCNO 12.9844 This project has been partially supported by a grant from the Spanish Government, project TIMOM (TIN2006-15265-C06-03).

References

[1] Uren ˜ a-L´opez, L.A., D ´ ıaz-Galiano, M.C. , Montejo-Raez , A. , and Mart´ın- Valdivia , M.T.: The Multimodal Nature of the Web: New Trends in Information Access. UPGRADE (The European Journal for the Informatics Professional) . Monograph: Next Generation Web Search , pp. 27 - 33 . 2007 .

[2] Quinlan , J.R. : Induction of Decision Trees Machine Learning, (1) , 81 - 106 . 1986 .

[3] Yang , Y. , and Pedersen , J.O.: A Comparative Study on Feature Selection in Text Categorization . Proceedings of ICML-97, 14th International Conference on Machine Learning . 1997 .

[4] Mitchell, T. : Machine Learning . McGraw Hill . 1996 .

[5] Lee , W. , and Xiang , D. : Information-Theoretic Measures for Anomaly Detection ( 2001 ). Proc. of the 2001 IEEE Symposium on Security and Privacy . 2001 .

[6] Oard , D.W. , Wang , J. , Jones , G.J.F. , White , R.W. , Pecina , P. , Soergel , D. , Huang , X. , and Shafran , I.: Overview of the CLEF-2006 Cross-Language Speech Retrieval Track . In Proceedings of the Cross Language Evaluation Forum (CLEF 2006 ), 2006 .

[7] Cover , T.M. , and Thomas , J.A. : Elements of Information Theory, Second Edition . WileyInterscience. July 2006

[8] Garc ´ıa- Cumbreras , M.A. , Uren˜ a-L´opez, L.A. , Mart´ınez- Santiago , F. , and Perea-Ortega , J.M. : BRUJA System . The University of Ja´ en at the Spanish task of QA@CLEF 2006 . In Proceedings of the Cross Language Evaluation Forum (CLEF 2006 ), 2006 .