English. This paper describes the first edition of the Question Answering for Frequently Asked Questions (QA4FAQ) task at the EVALITA 2016 campaign. The task concerns the retrieval of relevant frequently asked questions, given a user query. The main objective of the task is the evaluation of both question answering and information retrieval systems in this particular setting in which the document collection is composed of FAQs. The data used for the task are collected in a real scenario by AQP Risponde, a semantic retrieval engine used by Acquedotto Pugliese (AQP, the Organization for the management of the public water in the South of Italy) for supporting their customer care. The system is developed by QuestionCube, an Italian startup company which designs Question Answering tools.
Italiano. Questo lavoro descrive la prima edizione del Question Answering for Frequently Asked Questions (QA4FAQ) task proposto durante la campagna di valutazione EVALITA 2016. Il task consiste nel recuperare le domande piu` frequenti rilevanti rispetto ad una domanda posta dall’utente. L’obiettivo principale del task e` la valutazione di sistemi di question answering e di recupero dell’informazione in un contesto applicativo reale, utilizzando i dati provenienti da AQP Risponde, un motore di ricerca semantico usato da Acquedotto Pugliese (AQP, l’ente per la gestione dell’acqua pubblica nel Sud Italia). Il sistema e` sviluppato da QuestionCube, una startup italiana che progetta soluzioni di Question Answering. 1
Searching within the Frequently Asked Questions (FAQ) page of a web site is a critical task: customers might feel overloaded by many irrelevant questions and become frustrated due to the difficulty in finding the FAQ suitable for their problems. Perhaps they are right there, but just worded in a different way than they know.
The proposed task consists in retrieving a list of relevant FAQs and corresponding answers related to the query issued by the user.
Acquedotto Pugliese (AQP) developed a semantic retrieval engine for FAQs, called AQP Risponde1, based on Question Answering (QA) techniques. The system allows customers to ask their own questions, and retrieves a list of relevant FAQs and corresponding answers. Furthermore, customers can select one FAQ among those retrieved by the system and can provide their feedback about the perceived accuracy of the answer.
AQP Risponde poses relevant research challenges concerning both the usage of the Italian language in a deep QA architecture, and the variety of language expressions adopted by customers to formulate the same information need.
The proposed task is strongly related to the
one recently organized at Semeval 2015 and 2016
about Answer Selection in Community Question
Answering
The paper is organized as follows: Section 2 describes the task, while Section 3 provides details about competing systems. Results of the task are discussed in Section 4. 2
The task concerns the retrieval of relevant frequently asked questions, given a user query. For defining an evaluation protocol, we need a set of FAQs, a set of user questions and a set of relevance judgments for each question. In order to collect these data, we exploit an application called AQP Risponde, developed by QuestionCube for the Acquedotto Pugliese. AQP Risponde provides a back-end that allows to analyze both the query log and the customers’ feedback to discover, for instance, new emerging problems that need to be encoded as FAQ. AQP Risponde is provided as web and mobile application for Android2 and iOS3 and is currently running in the Acquedotto Pugliese customer care. AQP received about 25,000 questions and collected about 2,500 user feedback. We rely on these data to build the dataset for the task. In particular, we provide: a knowledge base of 406 FAQs. Each FAQ is composed of a question, an answer, and a set of tags; a set of 1,132 user queries. The queries are collected by analyzing the AQP Risponde system log. From the initial set of queries, we removed queries that contains personal data; a set of 1,406 pairs < query; relevantf aq > that are exploited to evaluate the contestants. We build these pairs by analyzing the user feedback provided by real users of AQP Risponde. We manually check the user feedback in order to remove noisy or false feedback. The check was performed by two experts of the AQP customer support.
2https://play.google.com/store/apps/ details?id=com.questioncube.aqprisponde& hl=it
3https://itunes.apple.com/it/app/ aqp-risponde/id1006106860
We provided a little sample set for the system development and a test set for the evaluation. We did not provide a set of training data: AQP is interested in the development of unsupervised systems because AQP Risponde must be able to achieve good performance without any user feedback. Following, an example of FAQ is reported: Question “Come posso telefonare al numero verde da un cellulare?” How can I call the toll-free number by a mobile phone? Answer “E` possibile chiamare il Contact Center AQP per segnalare un guasto o per un pronto intervento telefonando gratuitamente anche da cellulare al numero verde 800.735.735. Mentre per chiamare il Contact Center AQP per servizi commerciali 800.085.853 da un cellulare e dall’estero e` necessario comporre il numero +39.080.5723498 (il costo della chiamata e` secondo il piano tariffario del chiamante).” You can call the AQP Contact Center to report a fault or an emergency call without charge by the phone toll-free number 800 735 735...
Tags canali, numero verde, cellulare
For example, the previous FAQ is relevant for the query: “Si puo` telefonare da cellulare al numero verde?” Is it possible to call the toll-free number by a mobile phone?
Moreover, we provided a simple baseline based on a classical information retrieval model. 2.1
Data Format FAQs are provided in both XML and CSV format using “;” as separator. The file is encoded in UTF8 format. Each FAQ is described by the following fields: id a number that uniquely identifies the FAQ question the question text of the current FAQ answer the answer text of the current FAQ tag a set of tags separated by “,”
Test data are provided as a text file composed by two strings separated by the TAB character. The first string is the user query id, while the second string is the text of the user query. For example: “1 Come posso telefonare al numero verde da un cellulare?” and “2 Come si effettua l’autolettura del contatore?”. The baseline is built by using Apache Lucene (ver. 4.10.4)4. During the indexing for each FAQ, a document with four fields (id, question, answer, tag) is created. For searching, a query for each question is built taking into account all the question terms. Each field is boosted according to the following score question=4, answer=2 and tag=1. For both indexing and search the ItalianAnalyzer is adopted. The top 25 documents for each query are provided as result set. The baseline is freely available on GitHub5 and it was released to participants after the evaluation period. The participants must provide results in a text file. For each query in the test data, the participants can provide 25 answers at the most, ranked according by their systems. Each line in the file must contain three values separated by the TAB character: < queryid >< f aqid >< score >.
Systems are ranked according to the
accuracy@1 (c@1). We compute the precision of the
system by taking into account only the first
correct answer. This metric is used for the final
ranking of systems. In particular, we take into account
also the number of unanswered questions,
following the guidelines of the CLEF ResPubliQA Task
The system should not provide result for a particular question when it is not confident about the correctness of its answer. The goal is to reduce the amount of incorrect responses, keeping the number of correct ones, by leaving some questions unanswered. Systems should ensure that only the portion of wrong answers is reduced, maintaining as high as possible the number of correct answers. Otherwise, the reduction in the number of correct answers is punished by the evaluation measure for both the answered and unanswered questions.
Thirteen teams registered in the task, but only
three of them actually submitted the results for the
evaluation. A short description of each system
follows:
chiLab4It - The system described in
It is important to underline that all the systems adopt different strategies, while only one system (chiLab4It) is based on a typical question answer module. We provide a more detailed analysis about this aspect in Section 4. Results of the evaluation in terms of c@1 are reported in Table 1. The best performance is obtained by the chilab4it team, that is the only one able to outperform the baseline. Moreover, the chilab4it team is the only one that exploits question answering techniques: the good performance obtained by this team proves the effectiveness of question answering in the FAQ domain. All the other participants had results under the baseline. Another interesting outcome is that the baseline exploiting a simple VSM model achieved remarkable results.
A deep analysis of results is reported in
7https://github.com/SRvSaha/ QA4FAQ-EVALITA-16/blob/master/italian_ stopwords.txt 8Relevance is subjective.
We tried to mitigate issues related to relevance judgments by manually checking users’ feedback. However, this manual annotation process might have introduced some noise, which is common to all participants.
Regarding missing correct answers in the gold standard: this is a typical issue in the retrieval evaluation, since it is impossible to assess all the FAQ for each test query. Generally, this issue can be solved by creating a pool of results for each query. Such pool is built by exploiting the output of several systems. In this first edition of the task, we cannot rely on previous evaluations on the same set of data, therefore we chose to exploit users’ feedback. In the next editions of the task, we can rely on previous results of participants to build that pool of results.
Finally, in Table 2 we report some information retrieval metrics for each system9. In particular, we compute Mean Average Precision (MAP), Geometrical-Mean Average Precision (GMAP), Mean Reciprocal Rank (MRR), Recall after five (R@5) and ten (R@10) retrieved documents. Finally we report the success 1 that is equal to c@1, but without taking into account answered queries. We can notice that on retrieval metrics the baseline is the best approach. This was quite expected since an information retrieval model tries to optimize retrieval performance. Conversely, the best approach according to success 1 is the chilab4it system based on question answering, since it tries to retrieve a correct answer in the first position. This result suggests that the most suitable strategy in this context is to adopt a question answering model, rather than to adapt an information retrieval approach. Another interesting outcome concerns the system NLP-NITMZ.1, which obtains an encouraging success 1, compared to the c@1. This behavior is ascribable to the fact that the system does not adopt a strategy that provides an answer for all queries. 5
For the first time for the Italian language, we propose a question answering task for frequently asked questions. Given a user query, the participants must provide a list of FAQs ranked by relevance according to the user need. The collection 9Metrics are computed by the latest version of the trec eval tool: http://trec.nist.gov/trec_ eval/ baseline NLP-NITMZ.1 NLP-NITMZ.2 0.5190 of FAQs was built by exploiting a real application developed by QuestionCube for Acquedotto Pugliese. The relevance judgments for the evaluation are built by taking into account the user feedback.
Results of the evaluation demonstrated that only the system based on question answering techniques is able to outperform the baseline, while all the other participants reported results under the baseline. Some issues pointed out by participants suggest exploring a pool of results for building more accurate judgments. We plan to implement this approach in future editions of the task.
This work is supported by the project “Multilingual Entity Liking” funded by the Apulia Region under the program FutureInResearch.