The Acceptability and Complexity evaluation task for Italian (AcCompl-it) was aimed at developing and evaluating methods to classify Italian sentences according to Acceptability and Complexity. It consists of two independent tasks asking participants to predict either the acceptability or the complexity rate (or both) of a given set of sentences previously scored by native speakers on a 1-to-7 points Likert scale. In this paper, we introduce the datasets distributed to the participants, we describe the different approaches of the participating systems and provide a first analysis of the obtained results.
The availability of annotated resources and
systems aimed at predicting the level of
grammatical acceptability or linguistic complexity of a
sentence (see, among others,
The Acceptability & Complexity evaluation
task for Italian (AcCompl-it) at EVALITA 2020
We conceived AcCompl-it as a prediction task
where participants were asked to estimate the
average acceptability and complexity score of a set of
sentences previously rated by native speakers on
a 1-7 Likert scale and, if possible, to predict the
actual standard error (SE) among the annotations.
SE gives an estimation of the actual agreement
between human annotators: the highest the SE, the
lowest the agreement. The task is articulated in
three subtasks, as follows:
• the Acceptability prediction task (ACCEPT),
where participants have to estimate the
acceptability score of sentences (along with
their standard error); in this case, 1
corresponds to the lowest degree of acceptability,
while 7 corresponds to the highest level. The
assignment of a score on a gradual scale is in
line with the definition of perceived
acceptability that we intend to empirically inspect.
According to the literature, in fact,
acceptability is a concept closely related to
grammaticality but with some major differences
(see, among others,
The three subtasks were independent and participants could decide to participate in any one of them, though we encouraged participation in multiple subtasks, since the complexity metrics might be influenced by the grammatical status of an expression and vice versa. In line with this intuition, we distributed a subset of sentences annotated with both acceptability and complexity scores in order to investigate whether and to what extent there is a correlation between the two phenomena.
In all subtasks, participants were free to use external resources, and they were evaluated against a blind test set. 3 3.1
Acceptability dataset: it contains 1,683 Italian sentences annotated with human judgments of acceptability on a 7–point Likert scale. The number of annotations per sentence ranges from 10 to 85, with an average of 16.38. The dataset is constructed by merging the data of four psycholinguistic studies on minimal variations of controlled linguistic oppositions with different levels of grammaticality with a subset of 672 sentences generated from templates.
The first subset (128 sentences), taken from
riot b. La causa della rivolta sono le foto the cause of_the riot are the pictures del muro.
of_the wall This subset also contains declarative and interrogative (yes/no) sentences with a minimal verbal structure (contrasting preverbal vs postverbal subject position in unergatives (3a), unaccusatives (3b) and transitive predicates (3c)) (3) a. I cani hanno abbaiato j Hanno abbaiato i the dogs have barked j have barked the cani. dogs b. Gli autobus sono partiti j Sono partiti gli The buses have left j have left the autobus.
buses
c. Le bambine hanno mangiato il dolce j
the girls have eaten the dessert j
Hanno mangiato le bambine il dolce
have eaten the girls the dessert
The third set (320 sentences) is based on a study in
which number and person subject-verb agreement
and unagreement cases are tested
*write1Psg} a letter
The fourth one (48 sentences) contains
experimental items from
Whenever possible each sentence was also manually annotated according to the linguistictheoretic expectations for “grammaticality”, on a 4-points scale: * (ungrammatical, coded as 0), ?? (very marginal, coded as 0.66), ? (marginal, coded as 0.33) and OK (grammatical, coded as 1).
Complexity dataset: it comprises 2,530
Italian sentences annotated with human judgments of
perceived complexity on a 7–point Likert scale
as for the acceptability dataset. The number of
annotations per sentence ranged from 11 to 20,
with an average of 16.753. The corpus was
internally subdivided into two subsets
representative of two different typologies of data, i.e. 1,858
naturalistic sentences extracted from corpora and
672 artificially-generated sentences drawn from
the Acceptability dataset, and chosen to cover the
range of linguistic phenomena represented in its
templates. The first subset contains sentences
taken from the Universal Dependency (UD)
treebanks
For the purpose of the task, both datasets were split into training and validation samples with a proportion of 80% to 20%, respectively. 3.2
For the collection of judgments of sentence
acceptability and complexity by Italian native
speakers we relied on crowdsourcing techniques using
different platforms. More specifically, for
Acceptability, the set of sentences drawn from the
psycholinguistic studies described in Section 3.1
was annotated using an on-line platform based on
jsPsych scripts
Beyond complexity, the 672 artificiallygenerated sentences were also labelled for perceived acceptability according to the following question: “Quanto è accettabile questa frase da 1 (completamente agrammaticale) a 7 (perfettamente grammaticale)?” ‘How acceptable is this sentence from 1 (completely ungrammatical) to 7 (completely grammatical)?’
After collecting all annotations, we excluded workers who performed the assigned task in less than 10 minutes, which we set as the minimum threshold to accurately complete the survey. 3.3
Table 1 shows the average value, standard deviation and minimum and maximum score of complexity and acceptability labels for the whole dataset. As it can be noticed, complexity values are on average lower and less scattered than the acceptability ones. For this corpus, the lowest value on the Likert scale (1) – which should have been used to label sentences perceived as very easy, in line to the task question – was given only twice, specifically to the following sentences: (11) Dimmi il nome di una città finlandese. tell me the name of a town Finnish ‘Tell me the name of Finnish town’ (12) Quali uve si usano per produrre vino?
Which grapes PRT they_use to make wine? Conversely, for the acceptability corpus, the highest value on the Likert scale (i.e. 7, meaning in this case completely acceptable) was attributed to 26 sentences. For space reasons, we report here only two examples: (13) Le sorelle sono sopravvissute.
The sisters are survived.
‘the sisters have survived’ (14) I lupi hanno ululato.
The wolves have howled.
With respect to the ‘worst’ values, two sample sentences judged respectively as the most complex (i.e. 6.46 on the Likert scale) and (among) the least acceptable (1.55) in each dataset are the following ones, respectively: (15) Chi è che lui ha affermato che il professore who is that he has claimed that the professor aveva detto che lo studente avrebbe dovuto had said that the student hadsubj must considerare questo candidato? consider this candidate? (16) Il falegname è arrivato mentre noi The carpenter has arrived while we montavo la mensola.
were_assembling1Psg the shelf. min max
SCORE 3.12 1.04 1 6.46
SE 0.332 0.08
0 0.63
SCORE 4.45 1.7 1.13 7
SE 0.36 0.14
0 0.74
If we consider the internal composition of the two datasets we can see a more articulated picture depending on its various subparts (see Table 2 and 3). For complexity, average scores are higher for sentences created to display specific acceptability patterns, thus proving that acceptability does affect the perception of complexity. Note that the most complex sentence (reported in (15)) is contained in this set, and is ungrammatical (no gap).
Among the treebank sentences, those extracted from journalistic texts (ISDT_news) were judged on average as the most complex, questions as the easiest ones. Twitter and direct speech sentences obtained scores in between the highest and the lowest value and very close to each other. This is in line with stylistic and linguistic analysis showing that the language of social media inherits many features from spoken language.
For the whole acceptability dataset, the Spearman’s rank correlation coefficient between theoretically-driven grammaticality and mean acceptability labels is very strong (r(656)=.83, p<.001). While this could be somehow expected, when we focus only on the 672 sentences annotated for both complexity and acceptability, we still observe a significant but lower correlation between expected grammaticality and mean complexity (r(656)=.34, p<.001). Still considering this subset, an additional outcome is the moderate (and negative) correlation between the two metrics (r(672)=.49, p<.001), further suggesting that the more a sentence is perceived as complex, the less acceptable it is. 4
For both the ACCEPT and COMPL Task, the evaluation metric was based on Spearman’s rank correlation coefficient between the participants’ scores and the test set scores. For each task, two different ranks were produced according to the prediction of the relative scores and to standard errors. In each task a different baseline was defined: • in the ACCEPT task, it corresponds to the score assigned by a SVM linear regression using unigram and bigram of words as features; • in the COMPL task, it corresponds to the score assigned by a SVM linear regression using sentence length as its sole feature. 5
The AcCompl-it task received three submissions for each subtask from two different participants, for a total of 6 runs. Unfortunately, neither participant took part in the Open Task. Results for the other ones are reported in Tables 4 and 5.
The systems from the two participants in the
task follow very different approaches: one is based
on deep learning and trained on raw texts
Sarti’s systems obtained the best performance on both tasks using a similar multi-task learning (MTL) approach, which consists in leveraging the predictions of a state-of-the-art neural language model for Italian (i.e. UmBERTO3) fine-tuned on the two downstream tasks to augment the original development sets with a large set of unlabeled ex3https://github.com/musixmatchresearch/umberto
UmBERTO-MTSA (Sarti) ItVenses-run1 (Delmonte) ItVenses-run2 (Delmonte) Baseline amples extracted from available Italian treebanks. The bigger dataset was then split into different portions to train an ensemble of classifiers. The resulting MTL model was finally used to predict the complexity/acceptability labels on the original test sets.
Delmonte’s ItVenses system parses the sentences to obtain a sequence of constituents and a set of sentence-level semantic features (presence of agreement, negation markers, speech act and factivity). These features, along with constituent triples and their frequency in the training set and in the Venice Italian Treebank are weighed with various heuristics and used to derive a prediction. Agreement mismatches were checked using morphological analysis of verb and subject, while the argumental structure is inferred using a deep parser. The two versions of the system (run1 and run2) differ only in their use of features (run2 dispenses with proposional negation and certain verb agreement features).
As it can be seen, ItVenses’s performance were considerably lower than Sarti’s system (lower, in fact, than the baseline based on sentence length, in the COMPL prediction task). However, as better explained in the following section, in the artificial data subset, which has complex but far less diverse structures, the gap with the winning system is reduced in the COMPL task (cfr. Table 7) and, even more robustly, in the ACCEPT task (Table 6). 6
The extremely good performance of the winning
system in both tasks is not wholly unexpected in
light of the impressive results obtained by
current neural networks models across a variety of
NLP tasks. In this regard, it is worth noticing
that, in his report, the author compared the
performance of the best system based on multi-task
learning to the one obtained by a simpler version
of the UmBERTO-based model with standard
finetuning on the two downstream tasks, achieving
already very good results (.90 and .84 for
acceptability and complexity predictions on the training
corpus, respectively). Similarly, and especially
for the automatic assessment of sentence
acceptability, the scores obtained by the winning system
(.88) are in line with those reported in
To have a better understanding of the potential impact of different types of data on the predictive capabilities of the two systems, we further inspected the final results by testing each system on sentences representative of diverse linguistic phenomena and textual genres. To this end, we split the whole test set into the distinct subsets defined in the corpus collection process (cfr. Section 3.1) and we assessed the correlation score between predicted and real labels for each type: note that, for the ACCEPT predictions, this analysis was performed considering only two ‘macro–classes’, i.e. artificial vs psycholinguistics-related data, in order to have a significant number of examples in the test set. Similarly, for COMPL, we distinguished the artificially–generated sentences from sentences drawn from all treebanks. Results of this fine-grained analysis are shown in Tables 6 and 7.
Interestingly, although the gap between the two systems is still evident, we observed that artificial data have an opposite effect on their performance. In particular, as anticipated in the previous section, ItVenses is more accurate in predicting both the complexity and, especially, the acceptability level of this group of sentences. The opposite holds for Sarti’s system, which although still very good in both tasks, achieves lower correlation scores when tested against artificial data.
Running an exploratory analysis based on expected grammaticality, we observed that Sarti’s system performs much better in predicting the acceptability score on expected grammatical sentences (r=.80, p<.001) than on expected ungram
UmBERTO-MTSA (Sarti) 0.90** ItVenses-run1 (Delmonte) 0.42** ItVenses-run2 (Delmonte) 0.50**
UmBERTO-MTSA (Sarti) 0.74** ItVenses-run1 (Delmonte) 0.50** ItVenses-run2 (Delmonte) 0.46** 0.55** 0.24** 0.48** 0.33** 0.20* 0.25* matical ones (r=.76, p<.001). Similarly, but less robustly, the same numerical asymmetry is observed in both Delmonte’s runs: for grammatical predictions, RUN1 r=.33, RUN2 r=.35; for ungrammatical ones RUN1 r=.32, RUN2 r=.34, all correlations being equally significant (p<.001).
UmBERTO-MTSA (Sarti) 0.86** ItVenses-run1 (Delmonte) 0.25** ItVenses-run2 (Delmonte) 0.24**
UmBERTO-MTSA (Sarti) 0.70** ItVenses-run1 (Delmonte) 0.44** ItVenses-run2 (Delmonte) 0.51** SE 0.61** 0.13* 0.10* 0.06 -0.07 -0.11