This paper describes the design and reports the results of two questionnaires. The first of these questionnaires was created to collect information about the interest of industrial companies in the field of Italian text/speech analytics towards the evaluation campaign EVALITA; the second to gather comments and suggestions for the future of the evaluation and of its final workshop from the participants and the organizers of the campaign on the last two editions (2011 and 2014). Novelties introduced in the organization of EVALITA 2016 on the basis of the questionnaires results are also reported.
(a)
(b)
The EUROMAP analysis, dated back to 2003, detected structural limits in the Italian situation regarding
the Human Language Technology (HLT) market
Starting from the active private companies present in the aforementioned surveys, we created a
repository of enterprises working on Italian text and speech technologies in Italy and abroad. In order to find
new enterprises not listed in previous surveys, we took advantage of online repositories (e.g. AngelList2
and CrunchBase3) and of extensive searches on the Web. Our final list included 115 companies among
which 57 are not based in Italy. This high number of enterprises dealing with Italian also outside the
national borders, reinforces one of the findings of the 2014 Alta Plana survey
All the companies in our repository were directly contacted via email and asked to fill in the online questionnaire. After an introductory description and the privacy statement, the questionnaire was divided into three sections and included 18 questions. In the first section, we collected information about the company such as its size and nationality; the second had the aim of assessing the interest towards evaluation campaigns in general and towards a possible future participation in EVALITA. Finally, in the third section we collected suggestions for the next edition of EVALITA.
We collected responses from 39 private companies (response rate of 33.9%)5: 25 based in Italy (especially in north and central regions) and the rest in other 9 countries6. 27 companies work on text technologies, 2 on speech technologies and the remaining declares to do business in both sectors. The great majority of companies (84.6%) has less than 50 employees and, more specifically, 43.6% of them are start-up.
Around 80% of respondents thinks that initiatives for the evaluation of NLP and speech tools are useful
for companies and expresses the interest in participating in EVALITA in the future. Motivations behind
the negative responses to this last point are related to the fact that the participation to a campaign is
considered very time-consuming and also a reputation risk in case of bad results. In addition, EVALITA
is perceived as too academically oriented, too focused on general (i.e. non application-oriented) tasks
2https://angel.co/
3https://www.crunchbase.com/
4http://altaplana.com/TA2014
5This response rate is in line with the rates reported in the literature on surveys distributed through emails, see
6Belgium, Finland, France, Netherlands, Russia, Spain, USA, Sweden, and Switzerland. and with a limited impact on media. This last problem seems to be confirmed by the percentage of respondents who were not aware of the existence of EVALITA before starting the questionnaire, i.e. 38.5% with 24.1% among Italian companies.
For each of the questions regarding the suggestions for the next campaign (third section), we provided a list of pre-defined options, so to speed up the questionnaire completion, together with a open field for optional additional feedback. Participants could select more than one option. First of all we asked what would encourage and what would prevent the company from participating in the next EVALITA campaign. The possibility of using training and test data also for commercial purposes and the presence of tasks related to the domains of interest for the company have been the most voted options followed by the possibility of advertising for the company during the final workshop (for example by means of exhibition stands or round tables) and the anonymisation of the results so avoiding negative effects on the company image. On the contrary, the lack of time and/or funds is seen as the major obstacle.
Favorite domains and tasks for companies participating in the questionnaire are shown in Figure 1. Social media and news resulted to be the most popular among the domains of interest, followed by humanities, politics and public administration. Domains included in the “Other” category are survey analysis, financial but also public transport and information technology. For what concerns the tasks of interest, sentiment analysis and named entity recognition were the top voted tasks, but a significant interest has been expressed also about content analytics and Word Sense Disambiguation (WSD). In the “Other” category, respondents suggested new tasks such as dialogue analysis, social-network analysis, speaker verification and text classification. 3
The questionnaire for participants and organizers of past EVALITA campaigns was divided into 3 parts. In the first part respondents were required to provide general information such as job position and type of affiliation. In the second part we collected comments about the tasks of past editions asking to rate the level of satisfaction related to four dimensions: (i) the clarity of the guidelines,; (ii) the amount of training data; (iii) the data format; and (iv) the adopted evaluation methods. An open field was also available to add supplementary feedback. Finally, the third section aimed at gathering suggestions for the future of EVALITA posing questions on different aspects, e.g. application domains, type of tasks, structure of the final workshop, evaluation methodologies, dissemination of the results.
The link to the questionnaire was sent to 90 persons who participated in or organized a task in at least one of the last two EVALITA editions. After two weeks we received 39 answers (43.3% response rate) from researchers, Phd candidates and technologists belonging to universities (61.54%) but also to public (25.64%) and private (12.82%) research institutes. No answer from former participants affiliated to private companies was received.
Fifteen out of seventeen tasks of the past have been commented. All the four dimensions taken into consideration obtained positive rates of satisfaction: in particular, 81% of respondents declared to be very o somewhat satisfied by the guidelines and 76% by the format of distributed data. A small percentage of unsatisfied responses (about 13%) were registered on the quantity of training data and on the evaluation. In the open field, the most recurring concern was about the low number of participants in some tasks, sometimes just one or two, especially in the speech ones.
Respondents expressed the will to see some of the old tasks proposed again in the next EVALITA
campaign: sentiment polarity classification
The diverging stacked bar chart
Both questionnaires provided us with useful information for the future of EVALITA: they allowed us to acquire input on different aspects of the campaign and also to raise interest towards the initiative engaging two different sectors, the research community and the enterprise community.
Thanks to the questionnaire for industrial companies, we had the possibility to reach and get in touch with a segment of potential participants who weren’t aware about the existence of EVALITA or had little knowledge about it. Some of the suggestions coming from enterprises are actually feasible, for example by proposing more application-oriented tasks and by covering domains that are important for them. As for this last point, it is worth noting that the preferred domains are the same for both enterprises and former participants and organizers: this facilitate the design of future tasks based on the collected suggestions. Another issue emerged from both questionnaires is the need of improving the dissemination of EVALITA results in Italy and abroad, in particular outside the boarders of the research community.
The questionnaire for former participants and organizers gave us insights also on practical aspects related to the organization of the final workshop and ideas on how to change the systems evaluation approach taking into consideration different aspects such as replicability and usability.
The results of the questionnaires were presented and discussed during the panel “Raising Interest and Collecting Suggestions on the EVALITA Evaluation Campaign”7 organized in the context of the second Italian Computational Linguistics Conference8 (CLiC-it 2015). The panel has sparked an interesting debate on the participation of industrial companies to the campaign, which led to the decision of exploring new avenues for involving industrial stakeholders in EVALITA, as the possibility to call for tasks of industrial interest, that are proposed, and financially supported by the proponent companies. At the same time, the need for a greater internationalization of the campaign, looking for tasks linked to the ones proposed outside Italy, was highlighted. Panelists also wished for an effort in future campaigns towards 7http://www.evalita.it/towards2016 8https://clic2015.fbk.eu/ the development of shared datasets. Being the manual annotation of data a cost-consuming activity, the monetary contribution of the Italian Association for Computational Linguistics9 (AILC) was solicited.
The chairs of EVALITA 201610 introduced in the organization of the new campaign novel elements,
aimed at addressing most of the issues raised by both the questionnaires and the panel
EVALITA 2016 has an application-oriented task (i.e., QA4FAQ) in which representatives of three
companies11 are involved as organizers
For what concerns the internationalization issue, in the 2016 edition we had two EVALITA tasks
having an explicit link to other shared tasks proposed for English in the context of other evaluation
campaigns: the re-run of SENTIPOLC, with an explicit link to the Sentiment analysis in Twitter task
at SEMEVAL15, and the new NEEL-it, which is linked to the Named Entity rEcognition and Linking
(NEEL) Challenge proposed for English tweets at the 6th Making Sense of Microposts Workshop
We also used social media such as Twitter and Facebook, in order to improve dissemination of information on EVALITA, with the twofold aim to reach a wider audience and to ensure timely communication about various stages of the evaluation campaign.
As for the organization of the final workshop, a demo session is scheduled for the systems participating to the IBM challenge, as a first try to address the request from the community to have new participatory modalities of interacting with systems and teams during the workshop.
We are thankful to the panelists and to the audience of the panel ‘Raising Interest and Collecting Suggestions on the EVALITA Evaluation campaign’ at CLiC-it 2015, for the inspiring and passionate debate. We are also very grateful to Malvina Nissim and Pierpaolo Basile, who accepted with us the challenge to rethink EVALITA and to co-organize the edition 2016 of the evaluation campaign.