EVALITA1 is the evaluation campaign of Natural Language Processing and Speech Tools for Italian. Since 2007, the general objective of EVALITA is to promote the development and dissemination of language resources and technologies for Italian, providing a shared framework where different systems and approaches can be evaluated in a consistent manner. EVALITA is an initiative of the Italian Association for Computational Linguistics (AILC)2 and it is endorsed by the Italian Association for Artificial Intelligence (AI*IA)3 and the Italian Association for Speech Sciences (AISV)4.
iLISTEN - itaLIan Speech acT labEliNg. This task consists in automatically annotating dialogue
turns with speech act labels, i.e. with the communicative intention of the speaker, such as statement,
request for information, agreement, opinion expression, general answer
NLP4FUN - Solving language games. This task consists in designing a solver for “The Guillotine”
game, inspired by an Italian TV show. The game involves a single player, who is given a set of five
words - the clues - each linked in some way to a specific word that represents the unique solution
of the game. Words are unrelated to each other, but each of them has a hidden association with
the solution. Once the clues are given, the player has to provide the unique word representing the
solution. The participant systems are required to build an artificial player able to solve the game
SUGAR - Spoken Utterances Guiding Chef ’s Assistant Robots. This task goal is to develop a
voicecontrolled robotic agent to act as a cooking assistant. To this aim, a train corpus of spoken
commands is collected and annotated using a 3D virtual environment that simulates a real kitchen where
users can interact with the robot. The task specifically focuses on a set of commands, whose
semantics is defined according to the various possible combination of actions, items (i.e. ingredients),
tools and different modifiers
Open access to resources and research artifacts, such as data, tools, and dictionaries, is deemed crucial for the advancement of the state of the art in scientific research. Accessibility of resources and experimental protocols enable both full and partial replication of studies in order to further validate their findings, towards building of new knowledge based on solid empirical evidence. To foster reproducibility and encourage follow-up studies leveraging the resources built within EVALITA 2018, we introduced two novelties this year. First of all, we intend to distribute all datasets used as benchmark for the tasks of this edition. To this aim, we have set up a repository on Github5, in line with the good practices already applied by the organizers of the previous edition6. Also, the datasets for all the tasks will be 5The dataset of EVALITA 2018 made available by the task organizers can be found at: https://github.com/ evalita2018/data
6The datasets of EVALITA 2016 can be found at: https://github.com/evalita2016/data hosted and distributed by the European Language and Resources Association (ELRA). In addition, we decided to further encourage the sharing of resources by making availability of the systems an eligibility requirement for the best system award (see Section 4).
In the same spirit, we encouraged cross-community involvement in both task organization and
participation. We welcomed the initiative of the organizers of AMI, the Automatic Misogyny Identification
task
In the 2016 edition task organisers were encouraged to collaborate on the creation of a shared test
set across tasks
For the first time, this year we decided to award the best system across-task, especially that of young researchers. The award was introduced with the aim of fostering student participation to the evaluation campaign and to the workshop, and received a funding from Google Research, CELI7, and from the European Language and Resources Association (ELRA)8.
Criteria for eligibility, are (i) the availability of the system as open source software by the end of the evaluation period, when the results are due to the task organizers, and (ii) the presence of at least one PhD candidate, a master or a bachelor student among the authors of the final report describing the system. The systems will be evaluated based on: novelty, to be declined as novelty of the approach with respect to the state of the art (e.g. a new model or algorithm), or novelty of features (for discrete classifiers); originality, to be declined as identification of new linguistic resources employed to solve the task (for instance, using WordNet should not be considered as a new resource), or identification of linguistically motivated features; or implementation of theoretical framework grounded in linguistics; critical insight, to be declined as a deep error analysis that highlights limits of the current system and pave direction to future challenges; technical soundness and methodological rigor.
We collected 7 system nominations from the organizers of 5 tasks belonging to the Affect, Creativity and Style track and to the Hate Speech track. 14 students were involved in the development of the systems which received a mentions: 7 PhD students and and 7 master students. Most students are enrolled in Italian universities, but 5 of them. The award recipient(s) will be announced during the final EVALITA workshop, co-located with CliC-it 2018, the Fifth Italian Conference on Computational Linguistics9. 7https://www.celi.it/ 8http://elra.info/en/ 9http://clic2018.di.unito.it/it/home/
The tasks and the challenge of EVALITA 2018 attracted the interest of a large number of researchers from academia and industry, for a total of 237 single preliminary registrations. Overall, 50 teams composed of 115 individuals from 13 different countries participated to one or more tasks, submitting a total of 34 system descriptions.
A breakdown of the figures per task is shown in Table 1. With respect to the 2016 edition, we collected a significantly higher number of both preliminary registrations (237 registrations vs. 96 collected in 2016), teams (50 vs. 34 in 2016), and participants (115 10 vs. 60 in 2016), that can be interpreted as a signal that we succeeded in reaching a wider audience of researchers interested in participating in the campaign as well as a further indication of the growth of the NLP community at large. This result could be also positively affected by the novelties introduced this year to involve cross-community participation, represented by the ITAMoji and AMI tasks. Indeed, of the 50 teams that submitted at least one run, 12 include researchers from foreign institutions. In addition to this, this year all tasks have received at least one submission.
A further aspect of the success for this edition can be due to the tasks themselves, especially the
“Affect, Creativity and Style” and the “Hate Speech” tracks. Although these two tracks cover 60% of
all tasks, they have collected the participation of 82% of the teams (41 teams). This is clearly a sign of
growing interest in the NLP community at large in the study and analysis of new text types such as those
produced in Social Media platforms and (on-line) user-generated content, also reflecting the outcome of
the 2016 survey
Finally, we consider the new protocol for the submission of participants’ runs, consisting in three
nonoverlapping evaluation windows, as a further factor that may have positively impact the participation.
Indeed, from the 2016 survey, it emerges that the main reasons for not participating in the evaluation
either refer to personal issues or preferences (“I gave priority to other EVALITA tasks”) also due to the
difficulty of participating in the evaluation step of all tasks simultaneously, as the evaluation period was
perceived as too short to enable participation to more than one task
Finally, it is compelling to open a reflection on the distinction between constrained and unconstrained submissions and participation to the tasks. Half of the tasks, namely ABSITA, ITAMOji, IronITA, and AMI, paid attention to this distinction and the other half did not take it into account. In early evaluation campaigns, the distinction used to be very relevant as it aimed at distinguishing the contribution of features or the learning approach from external sources of information, mainly intended as lexical 10Please note that the unique participants that also submitted a report are 68. This drop is mainly due to the participation to more than one task, resulting in the submission of only one report from the same team. resources. In recent years, the spread and extensive use of pre-trained word embedding representations, especially as a strategy to initialize Neural Network architectures, challenges this distinction at its very heart. Furthermore, this distinction is also challenged by the development of multi-task learning architectures. A multi-task system could definitely represent an instance of an unconstrained system, although it exploits data from a different task, rather than a lexical resource or additional data annotated with the same information as that in the main task. As a contribution to the discussion on this topic, we think that proponents of tasks that aim at differentiating between constrained and unconstrained runs must specify what are the actual boundaries, in terms of extra training data, auxiliary tasks, use of word embeddings and lexical resources. 6
For this edition of EVALITA we introduced novelties towards supporting reproducibility and crosscommunity engagement, towards advancement of methodology and techniques for natural language and speech processing tasks beyond the performance improvement, which is typically used as a metrics to assess state of the art approaches in benchmarking and shared task organization. In particular, the decision to award the best-system across tasks is inspired by this vision and aim at emphasizing the value of critical reflection and insightful discussion beyond the metric-based evaluation of participating systems.
In line with the suggestion provided by the organizers of the previous edition in 2016
Finally, a new trend that has emerged this year is the presence of tasks, GxG and HaSpeeDe, that
aimed at testing the robustness of systems across text genres, further challenging the participants to
develop their system. This “extra challenge” aspect is a new trend in EVALITA that started with the
2016 SENTIPOLC task
We would like to thank our sponsors CELI11, Google Research and the European Language and Resources Association (ELRA)12 for their support to the event and to the best-system across task award. A further thank goes to ELRA for its offer and support in hosting the task datasets and systems’ results. We also thanks Agenzia per l’Italia Digitale (AGID)13 for its endorsement. 11https://www.celi.it/ 12http://elra.info/en/ 13https://www.agid.gov.it evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA’18), Turin, Italy.
CEUR.org.