English. This work describes a selfsupervised data augmentation approach used to improve learning models' performances when only a moderate amount of labeled data is available. Multiple copies of the original model are initially trained on the downstream task. Their predictions are then used to annotate a large set of unlabeled examples. Finally, multitask training is performed on the parallel annotations of the resulting training set, and final scores are obtained by averaging annotator-specific head predictions. Neural language models are finetuned using this procedure in the context of the AcCompl-it shared task at EVALITA 2020, obtaining considerable improvements in prediction quality.
Italiano. Questo articolo descrive un approccio di self-supervised data augmentation utilizzabile al fine di migliorare le performance di algoritmi di apprendimento su task aventi solo una modesta quantita` di dati annotati. Inizialmente, molteplici copie del modello originale vengono allenate sul task prescelto. Le loro previsioni vengono poi utilizzate per annotare grandi quantita` di esempi non etichettati. In conclusione, un approccio di multitask training viene utilizzato, con le annotazioni del dataset risultante in veste di task indipendenti, per ottenere previsioni finali come medie dei i punteggi dei singoli annotatori. Questa procedura e` stata utilizzata per allenare modelli del linguaggio neurali per lo shared task AcCompl-it a EVALITA 2020, ottenendo ampi miglioramenti nella qualita` predittiva.
In recent times, pre-trained neural language
models (NLMs) have become the preferred approach
for language representation learning, pushing the
state-of-the-art in multiple NLP t
Recently, pattern-exploiting training (PET,
Schick and Schutze (2020a,b) tackles the
dependence of NLMs on labeled data by first
reformulating tasks as cloze questions using task-related
patterns and keywords, and then using language
models trained on those to annotate large sets of
unlabeled examples with soft labels. PET can be
thought of as an offline version of knowledge
distillation
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
In this paper, I propose a simple self-supervised
data augmentation approach that can be used to
improve the generalization capabilities of NLMs
on regression and classification tasks for
modestsized labeled corpora. In short, an ensemble of
fine-tuned models is used to annotate a large
corpus of unlabeled text, and new annotations are
leveraged in a multi-task setting to obtain final
predictions over the original test set. The method
was tested on the AcCompl-it shared tasks of the
EVALITA 2020 campaign
Let:
L = [(x1; y1); : : : (xn; yn)] be the initial labeled corpus containing sentence-annotation pairs xi 2 X; yi 2 Yx. 1 U = [x01; : : : x0m] be a large unlabeled corpus such that m n M : xi ! y^i be a pre-trained neural language model with a single task-specific heads, taking sentence xi as input and predicting label yi at inference time.
For some k 2 N1, we begin by splitting L in k equal-sized segments L1; : : : ; Lk and finetune k identical versions of M using k-fold cross-validation. We call the resulting models M 1; : : : ; M k “NLMs with standard fine-tuning on the y target task”, with M i being trained on the subset L Li and evaluated on Li. Then, each sentence of U is passed to each model, obtaining the corpus
U 0 = [(x01; y^101 : : : y^10k); : : : ; (x0m; y^m01 : : : y^m0k)] (1) labeled with expert annotations from fine-tuned models. Predicted values are taken instead of 1yi can be either discrete or continuous in this context. probability distributions after the softmax, which are typically used in the knowledge distillation literature, to keep the approach simple while making it viable in the context of regression tasks.
Now that the large corpus is annotated, a
multitask NLM M T M : xi ! y_i1 : : : y_ik is fine-tuned on
U 0 by treating each annotation in the set y^01 : : : y^0k
as a separate task, using 1-layer feed-forward
neural networks as task-specific heads while
performing hard parameter sharing
As a final step, M T M is fine-tuned on a training portion of L, using as prediction scores f (y_i1 : : : y_ik), where f is a task and contextdependent aggregation function. For example, in the case of a classification task, one can select the majority vote from the ensemble of model heads as the final prediction, while in a regression setting this can be done by averaging scores across heads. Once fine-tuned, the model can be tested on the test portion of L using the same f as the aggregator. I refer to this approach as Multi-Task Self-Annotation (MTSA) in the following sections. 3
The ACCEPT and COMPL training corpora,
containing respectively 1339 and 2012
sentences labeled with average scores and
standard error across annotators, were used as
labeled datasets LA; LC . The two tasks were
learned separately, following the same
approach described in the previous section.
A set of multiple Italian treebanks
including train, dev, and test sets of the
Italian Stanford Dependency Treebank
Length (# of tokens) Length (characters) UmBERTo fine-tuned UmBERTo-STSA UmBERTo-MTSA
Length (# of tokens)
Length (characters)
UmBERTo fine-tuned
UmBERTo-STSA
UmBERTo-MTSA
-0.36
-0.39
-0.39
0.90
0.91
0.91
used both for fine-tuning M 1:::k during the
annotation part and for fine-tuning M T M .
The model is based on the RoBERTa
architecture
Since both tasks involve predicting both averaged scores and the original standard error across participants, the approach presented in the previous section was adapted to account for multi-task learning of scores and errors from the beginning, with each model M i producing both a predicted score y^0i and a predicted error ^0i for the annotation step. The k parameter was set to 5 to prevent excessive overlapping of training data across models, with the final multi-task model M T M : xi ! y_i1 : : : yi5; i1 : : : i5 returning prediction for scores and errors for all the five sets of fine-tuned model annotations.
Models M 1:::k were trained for a maximum of 15 epochs on the labeled training sets using early stopping (5 patience steps, 20 evaluation steps using a 10% slice as dev set), learning rate = 1e 5, batch size b = 32 and embedding dropout = 0:1. The model’s base variant was used, having a hidden size jhj = 768, and a maximum sequence length of 128. Notably, the representations at the last layer of the UmBERTo model were averaged to obtain a sentence-level representation instead of using the [CLS] token. During the training on the whole unlabeled corpus, the evaluation steps were increased to 100 to balance evaluation time with the corpus’s increased size. 4
UmBERTo surprisal: Sentence-level surprisal estimates are produced using the pretrained model without fine-tuning as:
m P (x) = Y P (wi jw1:i 1; wi+1:m) i=1 (2) Length (# of tokens): Length of the sentence in number of tokens Length (characters): Length of the sentence in number of characters (including whitespaces) UmBERTo fine-tuned: Predictions produced by Umberto with standard fine-tuning on complexity corpus annotations.
UmBERTo-STSA: A variant of the MTSA approach where instead of performing multitask learning over model annotations on U , we average them in a single score, and the model is trained on it with single-task finetuning.
UmBERTo-MTSA: The approach presented in this work.
From Table 1, it can be observed that, although length alone is already correlated with acceptability complexity scores, UmBERTo can leverage additional information from its representation to produce much stronger predictions. Interestingly, both the STSA and MTSA self-annotation approaches consistently outperform regular finetuning, especially for what concerns standard error scores. This fact suggests that self-annotation leads to better generalization capabilities in the model over downstream tasks when relatively few Acceptability
Complexity
UmBERTo-MTSA
annotations are available. While the contribution of multi-task learning is modest, the MTSA approach may prove especially beneficial when training models M 1:::k on scores produced by different annotators instead of using different folds of the same corpus, as in this case. In both cases, predicted surprisal scores act as poor predictors for downstream tasks. It should also be noted that length appears to be negatively correlated to acceptability scores (i.e. longer sentences are generally less acceptable), while the relation is positive in the case of complexity (i.e. longer sentences are generally more complex).
Table 2 reports the scores obtained by MTSA over the test sets for the ACCEPT and the COMPL shared tasks. The organizers’ baseline scores correspond to the correlation among gold labels and acceptability and complexity predictions produced by an SVM model trained on 1-grams and bigrams of sentences and an SVM trained on sentence length, respectively. The MTSA approach achieved the first rank in both tasks, with considerable improvements over baseline scores. 5
Finally, some error analysis is performed to gain additional insights on which factors influence the predictability of complexity and acceptability judgments. The Profiling-UD tool by Brunato et al. (2020a) is used to produce linguistic annotations on test sentences for both tasks. Given an input sentence, Profiling-UD produces roughly 100 numeric scores representing different phenomena and properties at different language levels.2 I then correlate the value of all features with y and , representing the mean absolute error between true and predicted values for scores and 2A description of produced annotations is omitted for brevity. Refer to Brunato et al. (2020a) for additional details. avg. score (y) std. error ( ) upos dist PROPN dep dist nmod avg max depth n prep chains prep chain len upos dist PRON dep dist root dep dist punct aux mood dist Imp n tokens avg links len max links len (y ) standard errors, respectively. Table 3 presents the results of the error analysis.
Strongly correlated values in Table 3 correspond to features that highly influence, either positively or negatively, the prediction capabilities of the MTSA model. Extreme task scores (avg. score), denoting either not very acceptable or highly complex sentences, are less predictable than their average counterparts by MTSA. Sentences for whose the standard deviation of scores is high across participants appear to be less predictable in the context of complexity scores, while this does not affect acceptability predictions.
Concerning acceptability, I found a significant correlation between acceptability prediction errors and the presence of multilevel syntactic structures, (avg max depth) multiple long prepositional chains (n prep chains, prep chain len) and nominal modifiers (dep dist nmod). From the complexity viewpoint, instead, the presence of inflectional morphology related to the imperfect tense in auxiliaries (aux mood dist Imp) was the only property related to higher prediction errors. However, high token counts (n tokens) and long dependency links (avg links len, max links len) were shown to make the variability in complexity scores more predictable.
Overall, results suggest that incorporating syntactic information during the model’s training process may further improve complexity and acceptability models.
This work introduced a simple and effective data augmentation approach improving the fine-tuning performances of NLMs when only a modest amount of labeled data is available. The approach was first formalized and then empirically tested on the ACCEPT and COMPL shared tasks of the EVALITA 2020 campaign. Strong performances were reported for both acceptability and complexity prediction using a multi-task self-training approach, obtaining the top position in both subtasks. Finally, an error analysis highlighted the unpredictability of extreme scores and sentences having complex syntactic structures.
The suggested approach, although computationally refined and well-performing, is lacking in terms of complexity-driven biases that may prove useful in the context of complexity and acceptability prediction. A possible extension of this work may include a complementary syntactic task (e.g., biaffine parsing, as in Glavas and Vulic (2020)) during multi-task learning to see if forcing syntactically-competent representations in the top layers may prove beneficial in the context of syntax-heavy tasks like complexity and acceptability prediction. Moreover, it would be interesting to evaluate multi-task learning performances with complexity and acceptability parallel annotations given the conceptual similarity between the two tasks and estimate the effectiveness of a feedforward network as the final aggregator f in the MTSA paradigm instead of merely averaging predictions. Finally, Du et al. (2020) findings suggest that using an unsupervised in-domain filtering approach may further improve the self-training procedure when large unlabeled corpora are available.
The author was supported by a scholarship for Data Science and Scientific Computing students from the International School of Advanced Studies (SISSA).