English. ChiLab4It is the Question Answering system (QA) for Frequently Asked Questions (FAQ) developed by the Computer-Human Interaction Laboratory (ChiLab) at the University of Palermo for participating to the QA4FAQ task at EVALITA 2016 competition. The system is the versioning of the QuASIt framework developed by the same authors, which has been customized to address the particular task. This technical report describes the strategies that have been imported from QuASIt for implementing ChiLab4It, the actual system implementation, and the comparative evaluations with the results of the other participant tools, as provided by the organizers of the task. ChiLab4It was the only system whose score resulted to be above the experimental baseline fixed for the task. A discussion about future extensions of the system is also provided.
Italiano. ChiLab4It e` il sistema di Question Answering (QA) usato per rispondere alle Frequently Asked Questions (FAQs), sviluppato dal Laboratorio di Interazione Uomo-Macchina (Chilab) dell’Universita` degli Studi di Palermo allo scopo di partecipare al task QA4FAQ nella competizione EVALITA 2016. Il sistema e` una versione del framework QuASIt, sviluppato dagli stessi autori e che e` stato ridefinito per il task in questione. Il report descrive le strategie che hanno consentito di realizzare ChiLab4It a partire da QuASIt, l’effettiva implementazione del sistema e le valutazioni comparative con gli altri team che hanno partecipato al task, cos`ı come sono state rese note dagli organizzatori. ChiLab4It e` stato l’unico sistema a superare la baseline sperimentale fissata per il task. Nella parte conclusiva del report, verranno altres`ı discussi i possibili sviluppi futuri del sistema.
This technical report presents ChiLab4It
The main characteristic of QuASIt is the underlying cognitive architecture, according to which the interpretation and/or production of a natural language sentence requires the execution of some cognitive processes over both a perceptually grounded model of the world (that is an ontology), and a previously acquired linguistic knowledge. In particular, two kinds of processes have been devised, that are the conceptualization of meaning and the the conceptualization of form.
The conceptualization of meaning allows to associate a sense to perceived forms, that are the words of the user query. A sense is the set of concepts of the ontology that explains the form; such a process is implemented considering the ontology nodes whose labels match best the forms from a syntactic point of view. The set of such nodes is the candidate sub-ontology to contain the answer to produce. The syntactic match is based on a syntactic measure.
The second process associates a syntactic expression to a meaning; it implements the strategies for producing the correct form of an answer, once it has been inferred. The form depends on the way QuASIt can be used, that is in both multiple choice and essay questions. In the case of multiple choice questions, the form must be one of the proposed answers. The system infers the correct answer among the proposed ones using the values of the properties’ ranges in the sub-ontology; the answer that better syntactically match such ranges is considered the correct answer. If no answer can be inferred in this way, a support text can be used if available. The support text can be either derived automatically by the system, using the plain text associated to the nodes of the sub-ontology (such as an abstract node in the DBPedia ontology2) or provided directly to the questions as in the case of a text comprehension task. In figure 1 the architecture of QuASIt is shown. The ontology and the linguistic knowledge are located respectively in the Domain Ontology Base and the Linguistic Base. The Mapping to Meanings (MtM) and the Mapping to Forms (MtF) modules are the components that model the cognitive processes related to the conceptualization of meaning and form respectively. The Unification Merging module is essentially the FCG engine (Steels, 2011) that is used to perform query parsing.
The strategy we implemented in ChiLab4It system is based on the QuASIt function that selects the correct answer to multiple choice questions using support text; the intuition was that a FAQ can be considered a support text that can be used for retrieving the more relevant FAQ to a user’s query. For this reason, in the next subsection, we describe this strategy in detail, and next we show how it was applied in the proposed tool.
Searching in a support text is a possible strategy to deal with unstructured information when an artificial agent is trying to answer a particular question. In this case the agent learns a possible answer by comprehending the text dealing with the question
topic. Such a process is implemented in QuASIt by the MtF module.
Formally, let Q = fq1; q2:::qng be the query of the user, and P = fp1; p2; :::pmg a sentence in the support text; each element in these sets is a token. P will be considered as much similar as Q when maximizing the following similarity measure m: m = j=j ( l + u) where = = fpj j 9qi 2 Q; J (pj ; qi) > g, and J (pj ; qi) is the Jaro-Winkler distance between a couple of tokens (Winkler, 1990). As a consequence, = Q \ P , and j=j is the number of matching tokens both in Q and P . l = 1 jjP=jj is the number of “lacking tokens” that are tokens belonging to Q that do not match in P , while u = 1 o(Q;=) is the number of “unj=j ordered tokens” that is the number of tokens in Q that do not have the same order in =; here o(a; b) is the function returning maximum number of ordered tokens in a with respect to b.
Both l and u are normalized in the range [0 : : : 1]; they are penalty values representing syntactical differences among the sentences. The higher u and l are, the lower is the sentences similarity. The and parameters weight the penalty, and they have been evaluated empirically through experimentation along with .
We re-used such strategy in ChiLab4It using different values for and parameters depending on which kind of support text we consider during the search, as next explained. 3
The basic idea of the proposed tool was to consider a FAQ as a support text. According to the provided dataset, a FAQ is composed by three textual fields: the question text, the answer text and the tag set. For each of these fields we applied the search strategy defined above; in particular we set different and parameters for each field in the m measure, depending on linguistics considerations. For this reason, we defined three different parameterized m measures named m1, m2 and m3. Moreover, further improvements were achieved by searching for the synonyms of the words of the query in the answer text. These synonyms were not considered in the QuASIt implementation.
Given the previously defined variables =, l and u, the and parameters were set according to the following considerations: question text; the and parameters are the same of QuASIt, that is = 0:1 and = 0:2. This choice is based solely on linguistic motivations; in fact, considering that the support text is a question such as the user query, both sentences to be matched will have interrogative form. As a consequence, both l and u influence the final match. The final measure is: answer text; the search is iterated for each sentence in the text. In this case, the and parameters are zero ( = 0 and = 0). This is because the answer text has a direct form, so the order of tokens must not be considered; moreover, a sentence in the answer text owns more tokens than the query, so this information is not discriminative for the final match.
In this case, the search is extended to the synonyms of the words in the query except to the synonyms of the stop-words; this extension has improved significantly the performances of the system. Empirical evaluations demonstrated that there were not the same improvements when the synonyms were considered for the other parts of a FAQ (question text and tag set) because in these cases the synonyms increase uselessly the number of irrelevant FAQs retrieved by the system.
Formally, let be the -expansion set
M = fmsi j msi = j=ijg where =i = fpj 2 Si\ j 9qk 2 Q; J (pj ; qk) > g The final similarity measure m2 will be the maximum value in M : m2 = max fmsi j msi = j=ijg tag set; the and parameters are zero ( = 0 and = 0) also in this case. This is because the tags in the set do not own a particular linguistic typology, so the information related to both the order of tokens and the lacking ones must not to be considered. As already explained, the synonyms are not included in this search. As consequence: m3 = j=j where = is the previously defined intersection among the query of the user and the set of tags.
A query will be considered as much similar as a FAQ when maximizing the sum of the measures defined previously, so the final similarity value is: mfaq = m1 + m2 + m3 These values were ordered, and the first 25 FAQs were outputted for a single query as required by the task. 3.1
In figure 2 the architecture of ChiLab4It is shown; the input is the query of the user, while the output is the list of the first 25 relevant FAQs. The sources became the FAQ base and the Wiktionary source from which the provided FAQ dataset and the synonyms are respectively queried.
The white module of such an architecture is the MtF module as implemented in QuASIt. The dark modules are the integrations that have been applied to the MtF module for customizing it to the FAQ domain; in particular, such integrations regard both the -expansion of the query and the setting of the analytic form (including parameters) of the m measure depending on the FAQ field. The first integration is implemented by the module, that returns the set for the query of the user retrieving the synset from Wiktionary3.
Parameters and the measure settings are performed by the FAQ Ctrl module which is encapsulated into the main MtF module; it retrieves the FAQ from the FAQ base and customizes the m measure according to the analyzed field (m1 for the question text, m2 for the answer text, m3 for the tag set). The MtF module computes such measures referring to the -expanded query, and finally the mfaq value is computed and memorized by the
FAQ Ctrl for tracing the id of the FAQ with the highest value. In this section we show a toy example with the aim of explaining better the searching process in the support text and how the similarity measure works. Such an example is a real question as retrieved in the data set provided by the organizers.
Let consider the query with id = 4, that is: “a quali orari posso chiamare il numero verde”.
In this case, the Q and the Sw set are: Q = fA; quali; orari; posso; chiamare; il; numero; verdeg and
Sw = fA; ilg being “a” and “il” the stop-words in the question.
The highest measure is computed by ChiLab4It in correspondence to the FAQ with id = 339, that is shown in table 1. Considering this FAQ, let compute the three measures for the question text, the answer text and the tag set.
In the first case the support text is the question text of the FAQ, and the P set is: P = fQuali; sono; gli; orari; del; numero; verdeg with jP j = 7. The m1 value will be computed considering that the intersection = between the question text and the query of the user is:
= = fquali; orari; numero; verdeg . The Jaro-Winkler distance is 1 for each word, and j=j = 4. Also, l = 1 jjP=jj = 1 47 = 0:428. hansweriIl servizio del numero verde assistenza clienti AQP 800.085.853 e attivo dal lunedi al venerdi dalle ore 08.30 alle 17.30, il sabato dalle 08.30 alle 13.00; il servizio del numero verde segnalazioni guasto 800.735.735 e attivo 24 ore su 24.h/answeri htagiinformazioni, orari, numero verdeh/tagi h/faqi For the calculation of u, we notice that o(Q; =) returns 4 because the tokens in Q are all ordered with respect to =, that means they follow the same sequence in =. As consequence, u = 1 o(Q;=) =
j=j 1 44 = 0. Substituting all values, m1 will be: clarity. The intersection =1 = \ S1 is simple =1 = fnumero; verde; orarig because these tokens have the highest Jaro-Winkler distance from the tokens in S1. As consequence, M = fj=1jg = f3g and m2 = 3.
In the third case, the support text is the tag set, so P = finf ormazioni; orari; numero; verdeg and = = forari; numero; verdeg. The m3 value is simply m3 = j=j = 3.
Finally, the m measure is computed adding the three calculated values, so m = 3:95 + 3 + 3 = 9:95 that represents the highest value among those computed for all FAQs in the dataset. The dataset used for the evaluation was the one provided by the QA4FAQ task organizers; they released such a dataset as a collection of both questions and feedbacks that real customers provided to the AQP Risponde engine.
In particular, such dataset includes: a knowledge base of about 470 FAQs, each composed by the text fields we referred to; In the next step, we consider the answer text; in the FAQ, this text is composed by only one a set of query by customers; sentence that becomes the new support text P , and a set of pairs that allows organizers to evalthe procedure will be applied once. In particular, uate the possible contestants. The orgaS = fS1g and P = S1 = fIl, servizio, del, numero, nizers analyzed the feedbacks provided by verde, assistenza, clienti,...., attivo, 24, ore, su, 24g as real customers of AQP Risponde engine, and shown in table 1. In this case, the m2 measure checked them for removing noise. depends only from the intersection between the
-expanded query and S1. In particular, the set Training data were not provided: in fact AQP is is computed unifying the difference set Q Sw = interested in the development of unsupervised sysfQuali; orari; posso; chiamare; numero; verdeg tems, like ChiLab4It is. with the synset from Wiktionary of each such According to the guideline, we provided results in token, so: = f[[quali], [orari], [posso], [chia- a text file purposely formatted, and for each query mare, soprannominare, chiedere, richiedere], in the dataset we considered the first 25 answers. [numero, cifra, contrassegno numerico, ma- However, only the first FAQ is considered relevant tricola, buffone, pagliaccio, elenco, gruppo, for the scope of the task. ChiLab4It is ranked acserie, classe, gamma, schiera, novero, taglia, cording to the accuracy@1 (c@1), whose formumisura, attrazione, scenetta, sketch, esibizione, lation is: ldgoiasngpc,einnossa,ag,toitmcoolaip,,iadm,oatteca,cghlriieaeqntutdaios,i,tofac],socn[itcvroearlsods,ee,gnppoua,nltliatdataol-,, c@1 = n1 (nR + nU nnR ) smorto, esangue, acerbo, giovanile, vivace, where nR is the number of correct answers, nU is vigoroso, florido, verdeggiante, lussureggiante, the number of unanswered questions, and n is the rigoglioso, agricolo, agrario, vegetazione, vigore, total number of questions. rigoglio, freschezza, floridezza, via, avanti, A participant could have provided two different ecologista, ambientalista, livido]]g, where the runs, but in our case we considered only the best synsets are represented in square brackets for configuration of the system. In table 2 we show the final results with the ranks of all participants as provided by the organizers; our tool performed better than the other participants, and it was the only one ranked above the experimental baseline. 5
ChiLab4It has been presented in this work, that is a tool designed for participating to the QA4FAQ task in the EVALITA 2016 competition. ChiLab4It relies on QuASIt, a cognitive model for an artificial agent performing question answering in Italian, already presented by the authors. QuASIt is able to answer both multiple choice and essay questions using an ontology-based approach where the agents manages both domain and linguistic knowledge.
ChiLab4It uses the functions of QuASIt aimed at answering multiple choice questions using a support text to understand the query because a FAQ can be regarded exactly as a support text, that can be used to understand the query sentence and to provide the answer. Moreover our tool enhances the sentence similarity measure introduced in our reference cognitive model in two ways. First, three separate measures are computed for the three parts of a FAQ that is question text, answer text and tag set, and they are summed to provide the final similarity. Second, the synonyms of the query words are analyzed to match the query against each sentence of the answer text of the FAQ to achieve linguistic flexibility when searching for the query topic inside each text.
ChiLab4It was tested with the competition data, and it resulted to be the winner having a c@1 rank well above the fixed experimental baseline. Future works are aimed at refining the development of the entire QuASIT system. Particular attention will be devoted in studying more refined versions of the similarity measure to take into account complex phrasal structures.