In the CLEF 2012 the BulTreeBank Group of LMD, IICT, BAS is participating for QA4MRE task for Bulgarian. The system represented in the paper exploits an NLP Pipeline for Bulgarian in order to process the questions, answers and the supporting texts. Then we represent the results of the analysis as a bag of linguistic units - lemmas, dependency relations. These bags of words are the match between the question plus answer and the sentences in the text. The answer that maximizes the overlap is selected as the correct one. Since the system is deterministic we have only one run. The score achieved by the run is 0.29. The other two runs are performed as baseline runs with randomly selected answers. Their scores are 0.20 and 0.12, respectively. Thus, the using of lin guistic units in the overlapping estimation provides significant improvements over the baseline.
Bulgarian language has been included into the set of participating languages in CLEF
tasks since 2004. It is the first time when Bulgarian systems have been tuned to the
new format of the CLEF main task, namely: Question Answering for Machine
Reading Evaluation (QA4MRE). For the previous formats, an infrastructure was designed
In this section we present the linguistic processing pipeline (BTB-LPP1) for Bulgarian which we used for analyzing of the data. BTB-LPP comprises three main modules: a Morphological Tagger, a Lemmatizer and a Dependency Parser. 2.1
The morphological tagger is constructed as a pipeline of three modules - two stat 1
The pipeline is developed on the basis of the language resources, created within BulTreeBank project. The prefix BTB stands for BulTreeBank. istical taggers trained on the Morphologically Annotated Part of BulTreeBank (Bul TreeBank-Morph)2 and a rule-based module exploiting a large Bulgarian Morphological Lexicon and manually crafted disambiguation rules.
SVM Tagger
The first statistical tagger uses the SVMTool
Rule-based Component
The task of this component is to correct some of the erroneous analyses made by
the SVM Tagger. The correction of the wrong suggestions is performed by two
sources of linguistic knowledge – the morphological lexicon and the set of context
based rules. In the process of repairing we used as much as possible from the
information provided by the SVM tagger. The context rules are designed in such a way that
they aim at achieving higher precision even at the cost of low recall. The lexicon
look-up is implemented as cascaded regular grammars within the CLaRK system –
Guided Learning System: GTagger
GTagger is based on the guided learning system -
The combination of the three components implements the morphological tagger of BTB-LPP. The SVM Tagger plays the role of a guesser for the unknown words. The rule-based component provides an accurate annotation of the known words, leaving some unsolved cases. GTagger provides the final result. This result is used by the lemmatizer and the dependency parser.
http://www.bultreebank.org/btbmorf/ 2.2
The second processing module of BTB-LPP is a functional lemmatization module, based on the morphological lexicon, mentioned above. The functions are defined via two operations on word forms: remove and concatenate. The rules have the following form: if tag = Tag then {remove OldEnd; concatenate NewEnd} where Tag is the tag of the word form, OldEnd is the string which has to be removed from the end of the word form and NewEnd is the string which has to concatenated to the beginning of the word form in order to produce the lemma. Here is an example of such a rule: if tag = Vpitf-o1s then {remove ох; concatenate а}
The application of the rule to the past simple verb form for the verb четох (remove: ох; concatenate: а) gives the lemma чета (to read). Additionally, we encode rules for unknown words in the form of guesser word forms: # ох and tag=Vpitf-o1s. In these cases the rules are ordered.
In order to facilitate the application of the rules, we attach them to the word forms in the lexicon. In this way, we gain two things: (1) we implement the lemmatization tool as a part of the regular grammar for lexicon look-up, discussed above and (2) the level of ambiguity is less than 2% for the correct tagged word forms. In case of ambiguities we produce all the lemmas. After the morphosyntactic tagging, the rules that correspond to the selected tags, are applied. 2.3
Many parsers have been trained on data from BulTreeBank. Especially successful was
the MaltParser of Joakim Nivre
Here is a table with the dependency tagset, related to the Dependency part of the BulTreeBank. This part has been used for training of the dependency parser: adjunct 12009 clitic 2263 comp 18043 conj 6342 conjarg 7005
Indirect Object (indirect argument of a non-auxiliary verbal head)
In addition to the dependency tags, also the morphosyntactic tags have been
attached to each word
Here is an example of a processed sentence. The sentence is Бразилия е епицентърът на пандемията на СПИН (Brazil is the epicenter of the AIDS pandemic.) After the application of the language pipeline, the result is represented in a table form following the CoNLL shared task format. It is given in Table 2.
The column WF corresponds to the order of the word forms in the sentence. The information in Ling column is the suffix of the corresponding tag (according to Bul TreeBank morphosyntactic tagset) after removing the prefix represented in column POSex (extended POS). The elements in Head point to number of the dependency head of the given word form. The Rel is the dependency relation between the two wordforms.
In the next section we present the procedure for using the pipeline for the QA4MR task for Bulgarian. 3
In the process of answer selection for each question we have performed the following steps: 1. The supporting texts were processed by the NLP pipeline described in the previous section; 2. For each question and each potential answer of the question we constructed a declarative sentence which provides evidence that the potential answer is really an answer of the question; 3. The analyses of the sentences in the texts are compared for similarity with the analysis of the declarative sentence produced in step 2. In this way we rank the answers for each question.
In the rest of the section we describe in more details each of the steps.
Each sentence in the texts was presented as a bag of linguistic units where each
unit is either a lemma, either a triple from the dependency tree for the sentence -
<DepLemma, Rel, HeadLemma>. In the triple DepLemma is the lemma of the dependency
node in the tree, HeadLemma is the lemma for the head node in the tree, Rel is the
relation between the nodes. Thus, the ranking of the answers will be done on the basis
of a sentence in the text and the bag of the selected linguistic units. The first decision
is motivated by the limitation of the current processing pipeline which cannot estab
lish reliable connections between the linguistic units in more than one sentence. The
second decision is motivated by the fact that the matching of dependency trees might
be very complicated, although we are aware of works on edit distance comparisons,
such as the one used in
In order to ensure the mapping between the analysis of the question-and-answer pair and the analyses of the sentences in the text we had to process the pair in the same way. The initial idea was to process the question and the corresponding answers separately, but there were some problems. The dependency parser is not good on fragments of sentences. But the answers are fragments in most cases. Some possible relations between the words in the question and in the answer had to be used. On the other hand, the ideal supporting sentence in the text would be would be composed of the question (potentially rearranged in a declarative form) and the answer in an appropriate way. Thus, we decided to convert each pair of a question and a potential answer into the best supporting declarative sentence. In our case this was done manually, but in future we envisage implementing this procedure automatically. Here are two examples:
Q1: Защо страдащите от деменция трябва да бъдат насърчавани да рисуват? (Why do the dementia sufferers have to be encouraged to take risks?) A1: защото това укрепва паметта, вниманието и възприемането
(because this would strengthen their memory, attention and perception) D1: Страдащите от деменция трябва да бъдат насърчавани да рисуват, защото това укрепва паметта, вниманието и възприемането.
(The dementia sufferers have to be encouraged to paint, because this would strengthen the memory, attention and perception.) Q2: Кой е епицентърът на пандемията на СПИН?
(Where is the epicenter of the AIDS pandemic?) A2: Бразилия
(Brazil) D2: Бразилия е епицентърът на пандемията на СПИН.
(Brazil is the epicenter of the AIDS pandemic.)
The ranking of each answer was done by calculating, first, the size of the intersection of the bag of linguistic units for each sentence in the texts and the bag for the pair's sentence. Then we calculated the maximum of the size of the intersections. This maximum was considered a rank of the pair question-and-answer and, thus, it is the rank of the answer. Then we selected the answer with the highest rank as an answer to the question. In case there is more than one answer with the same highest rank we selected randomly one of them.
For the type of questions for which we know possible variations of the realization
of the answers in the text
Three runs of the system have been performed. One is the actual application of the above procedure. We also performed two random runs in order to establish a baseline. The two baseline runs were evaluated with scores: 0.12 and 0.20. The actual run received a score 0.29. This score shows a significant improvement over the baseline scores.
The error analysis showed two main problems for the method. First, in many cases the words in the question-and-answer pair differ from the words used in the text. Second, we did not implement enough paraphrased linguistic units. 4
In the paper we presented a method for QA4MR for Bulgarian which exploits linguistic analyses of both - the question-and-answer pairs and the text. The similarity metric between the question-and-answer pairs and the sentences in the text is based on bag-of-linguistic units - lemmas and dependency relations between lemmas in the sentences. Our conclusion is that for improving the results, a good synonymic lexicon is needed to cover the lexical variety as well as the usage of more other knowledge re sources, such as the provided background collections, various kinds of thesauri and domain-specific dictionaries for the specialized terms. Additionally, we need to extend the paraphrases generation mechanism. In future, we will also work on the inclu sion of more semantic objects in the comparison algorithm using connections to ontological knowledge. Another restriction of the current method is that the comparison of the question-and-answer derived sentence is only with one sentence in the text. It is necessary to develop a better model that broadens the observations both - in the text and in the question-and-answer unit. We expect the approach, proposed here, would perform better on technical domains, where the degree of lexical variety is minimized and literal repetitions are used instead of synonyms.