IberLEF is a comparative evaluation campaign for Natural Language Processing Systems in Spanish and other Iberian languages. Its goal is to encourage the research community to organize competitive text processing, understanding and generation tasks in order to define new research challenges and set new state-of-the-art results in those languages. This paper summarizes the evaluation activities carried out in IberLEF 2022, which included 10 tasks and 19 subtasks dealing with sentiment, stance and opinion analysis, detection and categorization of harmful content, Information Extraction, Paraphrase Identification, and Question Answering. Overall, IberLEF activities were a remarkable collective efort involving 310 researchers from 24 countries in Europe, Asia, Africa, Australia and the Americas.
In this paper we summarize the activities carried on in IberLEF 2022, extracting some aggregated figures for a better understanding of this collective efort.
These are the ten tasks successfully run in 2022, grouped thematically:
ABSAPT [
PoliticES [
Rest-Mex [
DA-VINCIS [
DETESTS [
EXIST [
LivingNER [
PAR-MEX [
QuALES [
ReCoRES [
In terms of languages, the distribution per tasks (including subtasks) is shown in Figure 1. Spanish is, one more year, the central language of IberLEF (17 tasks) with Portuguese and English in a secondary role (2 tasks each). Main Spanish variants considered are those from Spain, Mexico, Uruguay and Perú.
In terms of abstract task types, the distribution of tasks can be seen in Figure 2. Out of a total of 19 tasks (each subtask is counted as a task here), the most popular type of task is multi-class classification (7 tasks), followed by sequence tagging and binary classification (4 each). There are also two ordinal classification tasks, two regression tasks, two KB linking tasks (one on entity linking and another one on taxonomy linking), two answer extraction tasks (one is multiple choice, which is also counted as classification, and the other one is span selection, which we also count as sequence tagging) and one text generation task. Interestingly, in 2022 there are four complex tasks which involve solving more than one core task at once (for instance, sequence tagging plus entity linking).
Compared with 2021, the trends are towards a less numerous (19 vs 29) but more diverse and more complex set of tasks, where binary classification is no longer the most popular type of task and several tasks imply solving many NLP problems at the same time.
In terms of evaluation metrics, the distribution can be seen in Figure 3, which depicts only
the main metrics used to rank systems in each task. As in previous years, there is a remarkable
predominance of F1 (11 tasks), even if it does not perfectly match the problem considered.
Accuracy is used by three tasks, MAE in two regression tasks, and there are other six metrics
that are used only in one task. Some of them correspond to the complex tasks which embed
subtasks (e.g. the mean of F1 scores for several tasks is used in one occasion, the mean of inverse
MAE and F1 scores for diferent tasks in other, or the average of F1 measures at diferent points
in the future with weights according to the time distance in other. The rest are Average Exact
Match [
Overall, in IberLEF as in other NLP competitive evaluation challenges we might still be relying too much on averages to combine diferent quality metrics: it has been common this year to combine F1 measures (which are harmonic averages) with other measures using some other form of averaging. This hides the actual behaviour of systems and give usually no clues on how to improve them. Also, again in 2022 the choice of metrics is, in general, barely justified, particularly in terms of how the system output is going to be used in realistic usage scenarios.
Finally, in terms of novelty/stability IberLEF 2022 has brought many new problems, with seven out of the 10 primary tasks being new this year. Only REST-MEX, EXIST and DETESTS had also been run in 2021.
In terms of types of textual sources, Figure 4 shows how they are used in IberLEF 2022 tasks. There is more diversity than previous years, with Twitter being less dominant: TripAdvisor reviews were used in 5 tasks, Twitter in 4, clinical cases in 3 subtasks (all belonging to the same task), exams, news comments and Gabs were used in two subtasks each, and finally news and gastronomy texts were used in one task each.
In terms of dataset sizes and annotation eforts, it is dificult to establish fair comparisons, because of the diversity of text sizes and the wide variance in terms of annotation dificulty. In any case, in the majority of cases (14 tasks) manually annotated datasets were below 6,000 instances. Two other tasks provided annotated collections comprising between 10,000 and 15,000 instances, and there was one task which provided over 40,000 annotated instances.
As for the reliability of the annotations, one useful indicator is inter-annotator agreement, which is reported in 9 out of 19 tasks. In the tasks where it is reported, annotator agreement is high in three cases and mid-low in another six. In general, mid-low agreement indicates the complexity of the task rather than poor annotation guidelines.
Overall the annotation efort in IberLEF 2022 keeps being a remarkable contribution to enlarge test collections for Spanish (and, less prominently, other languages). One more year, IberLEF has been carried out without specific funding sources (other than those obtained individually by the teams organizing and participating in the tasks). A centralized funding schema could certainly help reaching larger and better annotations in IberLEF as a whole.
In terms of progress with respect the state of the art, it is as usual dificult to extract aggregated conclusions for the whole IberLEF efort, in particular given the diversity of approaches for providing task baselines: in five tasks, no baseline was provided. In three, only a trivial baseline was included in the comparisons (e.g. majority class or random baselines in classification). Four tasks used SVM as baseline, and five used some variant of transformers (BETO in two occasions, BERT in another two and T5 in one). Only two used other types of baselines.
In the tasks that used baselines, the baseline was beaten (by a margin larger than 5%) by the best system in eight cases. In two cases, the diference was below 5% (one in favour of the best system, the other in favour of the baseline), and in the last two tasks, the baseline was better than any system. This is an indication that at least some of the tasks
In Figure 5 we display a pairwise comparison between the best system and the best baseline, for each of the tasks where at least one baseline is provided, and with respect to the oficial ranking metric used in each task. To avoid confusion, we have restricted the chart to tasks where the oficial metric varies between 0 (worst quality) and 1 (perfect output).
Given that IberLEF 2022 was not a funded initiative, participation has again been impressive, with a large fraction of current research groups interested in NLP for Spanish organizing and/or participating in one or more tasks. Overall, 310 researchers representing 169 research groups from 24 countries in Europe, Asia, Africa, Australia and the Americas were involved in IberLEF tasks1.
Figure 6 shows the distribution of research groups per country. This year, Mexico has the largest representation, with 54 groups, followed by Spain with 44 groups (note that all figures 1Statistics have been compiled from the submitted working notes, meaning two things: i) some groups and researchers may be counted twice if they have participated in more than one task; ii) real participation may be higher due to the number of teams who submitted runs but did not submit their working notes afterwards, and thus have not been counted in the statistics. reporting participation do not collapse duplicates: a group or a researcher participating in two tasks is counted twice).
Figure 7 shows the distribution of researchers (appearing as authors in the working notes) per country. The numbers are almost consistent with the distribution of groups per country, with some flips between USA and Brazil, or China, Chile and Vietnam. The top five, with Mexico, Spain, Brazil, USA, and China, represents roughly 80% of the researchers involved. The fact that there are two non-Spanish, non-Portuguese speaking countries in the top five, China and the USA, as well as others such as Vietnam or Canada in the top positions in terms of participation, indicates two things: first, that Spanish attracts the attention of the NLP community at large; and second, that current NLP technologies enable addressing diferent languages without language-specific machinery, other than pre-trained language models made available to the research community.
The distribution of research groups per task is shown in Figure 8. Participation ranges between 3 and 36 groups. As in other evaluation initiatives, participation seems to be driven not only by the task intrinsic interest, but also by the cost of entry: as usual, classification tasks (the most basic machine learning task, for which more plug and play software packages exist) receive more participation than tasks which require more elaborated approaches and more creativity to assemble algorithmic solutions.
In its third edition, IberLEF has again been a remarkable collective efort for the advancement of Natural Language Processing in Spanish and other Iberian languages, comprising 10 main tasks and 310 researchers involved, from institutions in 24 countries in Europe, Asia, Africa, Australia and the Americas. IberLEF 2022 has been one of the most diverse in terms of types of tasks and application domains, and has contributed to advance the field in the areas of sentiment, stance and opinion analysis, detection and categorization of harmful content, Information Extraction, Answer Extraction, and Paraphrase identification. In a field where machine learning is the ubiquitous approach to solve challenges, the definition of research challenges, the development of high-quality test collections that allow for iterative evaluation and the design of sound evaluation methodologies and metrics are perhaps the most critical aspects of research, and we believe IberLEF keeps making significant contributions to all of them.
This work has been partially supported by the Spanish Ministry of Science and Innovation, Project FairTransNLP (PID2021-124361OB-C32), and by CONACyT-México, Project CB-201501-257383. The work of the third author has been partially funded by CDTI under grant IDI-20210776, IVACE under grant IMINOD/2021/72, and grant PLEC2021-007681 funded by MCIN/AEI/10.13039/501100011033 and by European Union NextGenerationEU/PRTR.