This paper describes the rst participation of the SINAI (Intelligent Systems of Access Information) group of the University of JaØn in the AVE task of QA@CLEF 2007. We have developed a system made up of training and classi cation processes, that uses machine learning methods (bbr, timbl). Based on lexical features it obtains good results, a 41% of QA accuracy.
Our system is based on a machine learning method that makes use of a binary classi er to solve the answer validation. In our approach we can distinguish two processes applied to this classi er: training and classi cation.
In the training process we have extracted several features for all the used training collections1. Previous results have been evaluated using the existing entailment judgements of these collections, and ML parameters have been adjusted.
We have trained the classi er obtaining a learned model that will be used later in the classi cation process.
In the classi cation process we also extract the same features used in the training process for each pair question-answer. The classi cation algorithm uses these features and the learned model obtained in the training process. This algorithm returns a boolean value (correct or incorrect ) for each pair question-answer. Figure 1 describes the system architecture.
The extracted features are related to the lexical similarity. In our experiments we have applied di erent lexical similarity measures. All these features are explained below. 2.1
Lexical similarity
This experiment approaches the textual entailment task, based on the extraction of a set of lexical
measures, that check the existing similarity between the hypothesis-text pairs. Our approach is
similar to [
Before the calculation of the di erent measures, the rst step was to preprocess the pairs using the English stopwords list. Then, we have used the GATE2 architecture to obtain the stems of tokens. Once the stems have been obtained, we have applied four di erent measures or techniques: 1Answer Validation Exercise training collection and Third Recognizing Textual Entailment Challenge (RTE3) training.
2http://gate.ac.uk/
² Simple Matching: this technique calculates the semantic distance between the stems of
each question and its answer. If the distance exceeds a threshold, both stems are considered
similar and the similarity weight value increases in one. The accumulated weight is
normalized dividing it by the number of terms of the answer (hypothesis). In this experiment we
have considered the threshold 0.5. The values of semantic distance measure range from 0
to 1. In order to calculate the semantic distance between two stems, we have tried several
measures based on WordNet [
SIMmatching =
Pi2H similarity(i) jHj where H is the set that contains the elements of the answer (hypothesis) and similarity(i) is de ned like:
similarity(i) = ½ 10 iofth9ejr2wiTsesimL(i; j) > 0:5 ² Binary Matching: this measure is the same that the previous one, but modifying the similarity function: similarity(i) = ² Consecutive Subsequence Matching: this technique relies on forming subsequences of consecutive stems in the answer (hypothesis) and matching them in the question (text). The minimal size of the consecutive subsequences is two, and the maximum is the maximum size of the answer. Every correct matching increases in one the nal weight. The sum of the obtained weights of the matching between subsequences of a certain size or length is normalized by the number of sets of consecutive subsequences of the answer created for this length. These weights are accumulated and normalized by the size of the answer less one. The Consecutive Subsequence Matching technique is de ned in the following equations: where SHi is the set that contains the subsequences of the answer with i size or length and f(SHi) is de ned like: where
CSSmatching =
PjiH=2j f (SHi)
jHj ¡ 1 f (SHi) = matching(i) =
Pj2SHi matching(j)
jHj ¡ i + 1 ² Trigrams: this technique relies on forming trigrams of words in the answer and matching them in the question. If an answer trigram matches in question, then the similarity weight value increases in one. The accumulated weight is normalized dividing it by the number of trigrams of the answer.
The algorithms used in the experiments as binary classi ers are two: Bayesian Logistic Regression
(BBR)3 and TiMBL [
As it has been explained in previous sections, a model is generated via the supervised learning process. This model is used by the classi cation algorithm, which will decide whether an answer is entailed by the given snippet or not.
Table 1 shows two o cial results: where: ² Exp1 uses three features: three lexical similarities (SIMmatching + CSSmatching + Trigrams). The model has been trained using the devel data provided by the organization of the Pascal challenge, RTE-3, and the ML method used was BBR. comparison. ² Exp2 uses the same three features. The model has been trained using the devel data provided by the organization of the Answer Validation Exercise task, AVE-2007, and the devel data provided by the organization of the Pascal challenge, RTE-3. The ML method used was TiMBL.
As we expected, the best result is obtained by means of the use of both devel collections, RTE-3 and AVE-2007, and the use of TiMBL. We have to investigate why both results are too di erent. 4
In spite of the simplicity of the approach, we have obtained remarkable results: each set of features has reported relevant information, concerning to the entailment judgement determination. Our experiments approach the textual entailment task being based on the extraction of a set of lexical measures that show the existing similarity between the hypothesis-text pairs.
We have applied simple matching, Binary Matching and Consecutive Subsequence Matching, but the matching between pairs of words is relaxed by using the Lin’s similarity measure through Wordnet hierarchy.
Finally, we want to implement a hierarchical architecture based on constraint satisfaction networks. The constraints will be given by the set of available features and the maintenance of the integrity across the semantic interpretation process. 5
This work has been partially supported by a grant from the Spanish Government, project TIMOM (TIN2006-15265-C06-03), and the RFC/PP2006/Id_514 granted by the University of JaØn.
3http://www.stat.rutgers.edu/ madigan/BBR