In this paper we present an in-depth investigation of the linguistic knowledge encoded by the transformer models currently available for the Italian language. In particular, we investigate whether and how using different architectures of probing models affects the performance of Italian transformers in encoding a wide spectrum of linguistic features. Moreover, we explore how this implicit knowledge varies according to different textual genres.
In the last few years, the study of Neural
Language Models (NLMs) and their representations
has become a key research area in the NLP
community. Several methods have been devised
to obtain meaningful explanations regarding the
linguistic information encoded in NLMs
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). from higher layers are better for syntactic and semantic properties. Using a suite of probing tasks, Tenney et al. (2019a) deeply explore this behavior showing that the linguistic knowledge encoded by BERT through its 12/24 layers follows the traditional NLP pipeline.
While the vast majority of this research focused on English contextual representations, relatively little work has been done to understand the inner workings of non-English models. The study by de Vries et al. (2020) represents an exception in this context: authors apply the probing task approach to compare the linguistic competence encoded by a Dutch BERT-based model and multilingual BERT (mBERT), showing that earlier layers of mBERT are consistently more informative that earlier layers of the monolingual model. The survey by Nozza et al. (2020) also provides a comparative study of mBERT and language-specific BERT models but focused on the performance that each model obtains after training on several specific downstream tasks.
In this paper, we adopt a task-agnostic perspective to carry out an in-depth investigation of the linguistic knowledge implicitly encoded by 6 Italian monolingual models and multilingual BERT. We define a broad set of probing tasks, each corresponding to a specific property of sentence structure. We then compare the average performance reached by each model in predicting the feature value, evaluating the results obtained by models using their layer-wise sentence-level representations. A further comparative perspective, which to our knowledge is still rather under-investigated, concerns the study of how the architecture of the probing model itself influences probing scores. To address this point, for each model, we perform the same suite of probing tasks using both a linear SVR and a multilayer perceptron (MLP), and compare whether and how each probing task’s resolution is affected by the two architectures.
Since all experiments were carried out on
different sections of Italian Universal Dependency
Treebank
To inspect the inner knowledge of language
encoded by Italian Transformers, we relied on a suite
of 82 probing tasks, each of which corresponds
to predicting the value of a corresponding feature
modeling a specific property of the sentence. We
designed two sets of experiments. The first one
consists in comparing the linguistic knowledge
encoded by the Italian Transformers and evaluating
the best probing model for inferring such
knowledge from the NLMs. We compared the results
obtained with two simple probing models, a linear
SVR and a multilayer perceptron (MLP), which
take as input layer-wise sentence-level
representations extracted from the Italian models. These
representations are produced for each sentence of
different sections of the Italian Universal
Dependency Treebank (IUDT), version 2.5
Models and Data We relied on 7 pre-trained Italian Transformers models. Models statistics are reported in Table 1. 1https://github.com/dbmdz/berts 2Polignano et al. (2019) 3https://github.com/idb-ita/GilBERTo 4https://github.com/musixmatchresearch/umberto 5De Mattei et al. (2020) Name Multilingual-BERT BERT-base-italian1 AlBERTo2 GilBERTo3 UmBERTo-Commoncrawl UmBERTo-Wikipedia4 GePpeTto5
Training data BERT Architecture
Wikipedia
Wikipedia + OPUS (13GB)
OSCAR (71GB)
Wikipedia (7GB) GPT-2 Architecture
Wikipedia + ItWAC (14GB) (Baroni et al., 2009)
Sentence level representations were computed performing a Mean-pooling operation over the word embeddings provided by the models.
NLM’s linguistic competences are probed against five IUDT sections including texts representative of different textual varieties and genres. As shown in the overview in Table 2, we also distinguish the whole ISDT into different sub– corpora according to the specific language variety they represent, e.g. transcription of spontaneous speech (ISDT europarl), questions (ISDT quest) or simplified language (ISDT 2parole). The set of probing tasks consists of predicting the value of a specific linguistic feature automatically extracted from each POS tagged and dependency parsed sentence of the IUTD datasets.
The set of features is based on the ones described in Brunato et al. (2020) and are acquired from raw, morpho-syntactic and syntactic levels of annotation and can be categorised in 9 groups corresponding to different linguistic phenomena. As shown in Table 3, these features model linguistic phenomena ranging from raw text one, to morpho– syntactic information and inflectional properties of verbs, to more complex aspects of sentence structure capturing global and local properties of the whole parsed tree and of specific subtrees, such as the order of subjects and objects with respect to the verb, the distribution of UD syntactic relations, also including features referring to the use of subordination and to the structure of verbal predicates.
All these features have been shown to play a
highly predictive role when leveraged by
traditional learning models on a variety of
classification problems, covering different aspects of
stylometric and complexity analysis. In addition, in
their recent work, Miaschi et al. (2020) showed
that these features can be effectively used to
profile the knowledge encoded in the language
representations of a pretrained NLM, specifically the
English Bert, and how it changes across layers.
Since these features are based on the UD
formalism, which guarantees the comparative
encoding of language phenomena between the two
languages
We first investigate which is the best architecture for probing the linguistic knowledge encoded by the Italian Transformers. Since many of our probing features are strongly related to sentence length, we compared the two probing models’ results with the ones obtained by a baseline corresponding to a LinearSVR model trained using only sentence length as input feature. Table 4 reports average R2 results6 for all the 7 NLMs obtained with the LinearSVR and the MLP probing models, along with baseline scores. The MLP probe is a three-layer feedforward network with ReLU activations and was selected to investigate the presence of nonlinear relations in representations, which could hamper the probing performance of the LinearSVM probe, but would be highlighted by a sharp difference between MLP and LinearSVM performances. As a first remark, we notice that both probing models outperform the baseline. This proves that all NLMs encode a spectrum of phenomena that, although related to sentence length, require a more sophisticated linguistic knowledge to be accurately predicted. Best scores are obtained with the MLP model, which achieved higher R2 scores especially for features grouping more complex syntactic phenomena (e.g. TreeStructure, SyntacticDep). Interestingly enough, the LinearSVR model outperforms the MLP by more than .30 R2 points when predicting features related to vocabulary richness (Vocabulary).
In order to ensure that our probes are actually showing the linguistic generalization abilities of the NLMs rather than learning the linguistic tasks, we also tested the probing models using the control task approach devised in Hewitt and Liang (2019). We produced a control version of the IUDT corpus by randomly shuffling the linguistic features assigned to each sentence and performed the same probing tasks with the two probing classifiers for all NLMs representations. The correla6The Coefficient of determination (R2) is a statistical measure of how close the data are to the fitted regression line and corresponds to the proportion of the variance in the dependent variable that is predictable from the independent variable(s). tion and R2 scores between regressors’ predictions and shuffled scores were low (< 0:05) and comparable for both the SVR and the MLP. These results support the claim that NLMs representations encode information closely related to linguistic competence and that our probing models are not relying on spurious signals unrelated to our linguistic properties to solve the regression task.
To investigate how each transformer encodes the linguistic knowledge, we report in Figure 1 average R2 scores obtained with the two probing models for all the 7 NLMs. As we can notice, the seven transformers achieve quite similar results when considering all features as a whole, although BERT-base-italian has the best overall performance (0.65 for all features). The same did not hold when we analyzed their performances in terms of R2 scores for the different previously described groups of features. For instance, we can notice that, for both the probing models, features related to the distribution of syntactic relations (SyntacticDep) are better predicted by GePpeTto, while GilBERTo and UmBERTo-Commoncrawl are the best ones in the prediction of tree structure properites. Differences hold for what regards competencies related to vocabulary richness (Vocabulary): while UmBERTo-Wikipedia extensively outperforms all the other transformers using the MLP model, the best transformer is BERT-baseitalian when these competences are probed with the LinearSVR model.
Similar trends can be observed in Figure 2, where we report how the linguistic knowledge encoded by the 7 NLMs evolves across layers according to the two probing models. Regardless of the architectures, for all transformers, raw text features (RawText) are mainly encoded in the first layers, while the knowledge about the order of subject/object (Order) and the use of subordination (Subord) increases consistently across layers and specifically in the first ones. Contrarily to what was observed by de Vries et al. (2020), mBERT’s linguistic knowledge is not encoded systematically earlier than in monolingual transformers. This perspective of analysis also reveals other differences among the considered transformers: e.g. even though GePpeTto has a lower average competence on verb inflection (see Figure 1), it achieves the highest scores in the middle layers. Focusing instead on differences between layerwise scores obtained by the two probing models, we can clearly notice that the encoding of linguistic knowledge shows a quite rough trend for what concerns the results obtained with the MLP. This is particularly the case of features belonging to the vocabulary, POS and tree structure groups.
Finally, we inspected whether the overall linguistic competence encoded in the contextual representations of each model changes according to the type of texts in the different IUDT sections we considered. As we could expect, the results reported in Figure 3 show that all transformers achieve lower performance when they have to predict the value of features extracted from treebanks representative of social media language (PoSTWITA and TWITTIRO` ). Quite surprisingly, it is also the case of AlBERTo which is trained on Twitter data. A possible explanation is that, although PoSTWITA and TWITTIR O` contain sentences representative of Twitter language, these sentences are still quite close to the Italian standard language, in order to be compliant with the UD morpho-syntactic and syntactic annotation schema. On the contrary, AlBERTo’s training set is derived from Twitter’s official streaming API that included all possible typologies of sentences. However, bert-base italian is slightly less affected by the non-standard linguistic peculiarities of this genre. Similarly to what is observed for the whole Italian dataset (see Figure 1), this model also reaches the highest performance in almost all different IUDT sections, except for the one containing interrogative sentences (isdt quest). Interestingly, this type of sentence is hardly mastered by all models. This is possible due to the fact that interrogative sentences are more likely to display a less canonical distribution of morphosyntactic and syntactic phenomena, hence being more difficult to encode effectively. 4
In this paper we presented an in-depth comparative investigation of the linguistic knowledge encoded in the Italian transformer models. Relying on a suite of more than 80 probing features and testing our approach with two different probing models, we showed that MLP is the best model for inferring the amount of information implicitly encoded in the NLMs representations. We also observed that BERT-base-italian achieved best scores in average, but the linguistic generalization abilities of the examined transformers vary according to specific groups of linguistic phenomena and across layers. Finally, we examined how the linguistic knowledge learned by the NLMs is affected by the distinct textual varieties available in Italian treebanks showing, for instance, that social media language represents a harder domain for all models.
We are currently investigating if the linguistic knowledge encoded by a NLM positively affects the resolution of downstream tasks, as already suggested by the recent work by Miaschi et al. (2020) for English. This connection, which is still rather investigated, can improve our understanding of how such models make their decisions.