We investigate to what degree existing LLMs encode abstract linguistic information in Italian in a multi-task setting. We exploit curated synthetic data on a large scale - several Blackbird Language Matrices (BLMs) problems in Italian - and use them to study how sentence representations built using pre-trained language models encode specific syntactic and semantic information. We use a two-level architecture to model separately a compression of the sentence embeddings into a representation that contains relevant information for a task, and a BLM task. We then investigate whether we can obtain compressed sentence representations that encode syntactic and semantic information relevant to several BLM tasks. While we expected that the sentence structure - in terms of sequence of phrases/chunks - and chunk properties could be shared across tasks, performance and error analysis show that the clues for the diferent tasks are encoded in diferent manners in the sentence embeddings, suggesting that abstract linguistic notions such as constituents or thematic roles does not seem to be present in the pretrained sentence embeddings. L'obiettivo di questo lavoro è indagare fino a che punto gli attuali LLM apprendono rappresentazioni linguistiche astratte in configurazioni multitask. Utilizzando dati sintetici curati su larga scala di vari problemi BLM in italiano, studiamo come le rappresentazioni di frasi costruite da modelli di linguaggio pre-addestrati codifichino le informazioni semantiche e sintattiche. Abbiamo utilizzato un'architettura a due livelli per modellare separatamente, da un lato, la compressione degli embeddings delle frasi di input in una rappresentazione che contiene informazioni rilevanti per i tasks BLM e, dall'altro lato, il BLM stesso. Abbiamo poi verificato se fosse possibile ottenere rappresentazioni compresse di frasi che codificano informazioni sintattiche e semantiche rilevanti per i diversi tasks BLM. Contrariamente alla predizione che la struttura della frase - in termini di sequenza di frasi/chunks - e le proprietà dei chunk possano essere condivise tra i vari tasks, i risultati e l'analisi degli errori mostrano che gli indizi per i diversi task sono codificati in modo diverso negli embeddings delle frasi. Questo risultato suggerisce che nozioni linguistiche astratte come i costituenti o i ruoli tematici non vi sembrano essere presenti.
tic structure and argument structure – can be assembled
from the information encoded in the sentence
embedDriven by increasing computational scale and progress dings. This, however, may not be due to a deeper
unin deep learning techniques, NLP models can rival hu- derstanding of such information encoded by LLMs, but
man capabilities on established benchmarks. New bench- rather because of useful surface indicators [
Blackbird’s Language Matrices (BLM) [
Attribution 4.0 International (CC BY 4.0).
We make two contributions: (i) an initial core BLM dataset for Italian that covers linguistic problems of diferent nature; (ii) single and multi-task experiments that provide new insights into the information encoded by LLMs.
The datasets are available at https://www.idiap.ch/datas et/(blm-agri|blm-causi|blm-odi) and the code at https: //github.com/CLCL-Geneva/BLM-SNFDisentangling.
BLM agreement problem (BLM-AgrI)
Context Template NP-sg PP1-sg NP-pl PP1-sg NP-sg PP1-pl NP-pl PP1-pl NP-sg PP1-sg NP-pl PP1-sg NP-sg PP1-pl
systems’ performance by using knowledge shared across PP1-sg multiple tasks [9]. PP2-pl
Multi-task learning architectures include parallel, hier- PP2-pl archical, and modular designs [10]. Parallel architectures PP2-pl share intermediate layers across tasks, conducive to efi- PP2-sg cient knowledge transfer [11]. Hierarchical architectures Figure 1: BLM instances for verb-subject agreement, with capture task dependencies by layering task-specific mod- two attractors. We build candidate answers displaying one ules on shared bases. Modular approaches selectively of two types of errors: (i) sequence errors: WNA= wrong nr. share components among tasks to balance between gen- of attractors; WN1= wrong gram. nr. for 1 attractor noun eralisation and task-specific optimisation [ 12]. These (N1); WN2= wrong gram. nr. for 2 attractor noun (N2); training strategies are not mutually exclusive and can be (ii) grammatical errors: AEV=agreement error on the verb; combined. AEN1=agreement error on N1; AEN2=agreement error on N2.
Multi-task learning can be used eficiently in resourceconstrained environments, to counter data scarcity and 3. The BLM task and the BLM overfitting: aggregating training data and sharing param- Italian datasets eters across related tasks acts as a form of data augmentation [13]. Raven’s progressive matrices are multiple-choice com
Efective multi-task learning depends on the related- pletion IQ tests, whose solution requires discovering
unness of the tasks involved. Tasks that are similar or have derlying generative rules of a sequence of images [
Pretrained large language models exhibit general- plicitly illustrates these grammatical properties. This purpose abilities and knowledge, with high results with sequence also follows some extra-linguistic progression little or no fine-tuning on downstream tasks [ 17, 18]. We rules. Each context is paired with a set of candidate ancan then regard these language models as the results of swers. The answer sets contain minimally contrastive "multi-task" learning, and our aim here is to test whether examples built by corrupting some of the generating sentence embeddings obtained from these models en- rules. code syntactic and semantic information consistently, The BLM Italian datasets consists of BLMs focused such that diferent BLM problems that rely on similar on the property of subject-verb agreement and two linguistic information draw on the same clues from these transitive-intransitive alternations: the change-of-state representations. In particular, we will use BLM tasks on alternation and the object-drop alternation. subject-verb agreement – which relies on chunk structure and the chunks’ grammatical number properties – and on verb alternations – which relies on chunk struc- 3.1. BLM-AgrI – subject-verb agreement ture and the chunks’ semantic role properties – to test in Italian whether chunk structure is encoded in a manner that allows for it to be shared by the two tasks.
The BLM-AgrI dataset is created by manually translating
the seed French sentences [
matical property, agreement, the Causative (Caus) and Object-drop (Od) alternation datasets test lexical semantic properties of verbs, their ability to enter or not a causative alternation. Caus represents the causative/inchoative alternation, where the object of the transitive verb bears the same semantic role (Patient) as the subject of the intransitive verb (L’artista ha aperto la finestra/La finestra si è aperta ‘The artist opened the window’/‘The window opened’). The transitive form of the verb has a causative meaning. In contrast, the subject in Od bears the same semantic role (Agent) in both the transitive and intransitive forms (L’artista dipingeva la ifnestra/L’artista dipingeva ‘the artist painted the window’/‘the artist painted’) and the verb does not have a causative meaning [19, 20].
BLM-CausI context and answers The context set of the verb alternation varies depending on the presence of one or two arguments and their attributes (agents, Ag; patients, Pat) and the active (Akt) and passive (Pass) or passive voice of the verb. The non-linguistic factor that structures the sequence is an alternation every two items between a prepositional phrase introduced by any preposition (e.g., in pochi secondi, P-NP) and a PP introduced by the agentive da-NP (e.g., dall’artista, da-Ag/da-Pat).
The answer set is composed of one correct answer
and contrastive wrong answers, all formed by the same
four elements: a verb, two nominal constituents and a
prepositional phrase. Figure 2 shows the template.1
BLM-OdI Context and Answers The BLM for Od is
the same as for Caus, but here the passive voice serves
as a confounding element and one of the contrastive
answers for Caus is, in fact, the correct answer here.
1Following BLM formal specifications [
Caus context Caus answers 1 Ag Akt Pat P-NP 1 Pat Akt da-NP Correct 2 Ag Akt Pat da-NP 2 Ag Akt da-NP I-Int 3 Pat Pass da-Ag P-NP 3 Pat Pass da-Ag ER-Pass 4 Pat Pass da-Ag da-NP 4 Ag Pass da-Pat IER-Pass 5 Pat Pass P-NP 5 Pat Akt Ag R-Trans 6 Pat Pass da-NP 6 Ag Akt Pat IR-Trans 7 Pat Akt P-NP 7 Pat Akt da-Ag E-WrBy ? ??? 8 Ag Akt da-Pat IE-WrBy
Od context Od answers 1 Ag Akt Pat P-NP 1 Pat Akt da-NP I-Int 2 Ag Akt Pat da-NP 2 Ag Akt da-NP Correct 3 Pat Pass da-Ag P-NP 3 Pat Pass da-Ag IER-Pass 4 Pat Pass da-Ag da-NP 4 Ag Pass da-Pat ER-Pass 5 Pat Pass P-NP 5 Pat Akt Ag IR-Trans 6 Pat Pass da-NP 6 Ag Akt Pat R-Trans 7 Ag Akt P-NP 7 Pat Akt da-Ag IE-WrBy ? ??? 8 Ag Akt da-Pat E-WrBy
The training and testing are disjoint (agreement data is split based on the correct answer, the alternations data based on the verb). Agreement has 230 test instances for type I, 4121 for types II and III. The verb alternations have 240 test instances for all subsets. We randomly sample a number of training instances, depending on the experimental set-up.
Sentence embeddings encode much information from
the input sentence – lexical, syntactic, semantic, and
possibly other types of information. Previous experiments
have shown that sentence embeddings can be compressed
into very small representations (vectors of size 5) that
encode information about the structure of the sentence
in terms of chunks and their properties, such that they
contribute to finding the sequence patterns in BLMs [
In this work, we investigate whether several BLM tasks can share the same structural information from a sentence embedding. Towards this end, we built a multi-task version of a two-level system, illustrated in Figure 3. In this system, one level processes individual sentences and learns to compress them into small vectors that retain information pertinent to a task and the other level uses the compressed sentence representation to find patterns across an input sequence to solve a BLM task. The multitask variation consists in a single shared sentence-level component, and multiple task components, one for each of the BLM tasks.
The BLM problems encode a linguistic phenomenon () = (ˆ, +, − ) through data that has structure on multiple levels – + (|| (0, 1)) within sentences, and across a sequence of sentences.
We can exploit this structure to develop an indirectly (ˆ, +, − ) =
supervised approach to discover and use these diferent (0, 1 − (ˆ, +)
lteavskelsasofasttwruoc-tsuterep. pWroectehsuss: (mi)ocdoemltphreesssolivnidnigviodfuaalBsLeMn- + ∑︀− ∈−(^,− ) )
tences into a representation that emphasizes the sentence
structure relevant to the BLM problem (e.g. chunks and
their grammatical number for the subject-verb agreement
task) (ii) use the compressed representations to detect
the sequence-level pattern and solve the BLM task. This
two-step process has been shown to be used by people
solving visual intelligence tests [21]. In our case, this () = (, , ∖ {})
setup allows us to investigate whether the sentence level + ( | (0, 1)).
can be guided to learn shared information, relevant to The loss of the two-level systems is:
the diferent linguistic tasks described in section 3.
twWineedimarpchleimtecetnutrethiilsluastprparteodacihn Finigtuhree 3tw,aon-dledveeslcirnibteerd- () = ∑︀∈ () + ()
in detail elsewhere [
computed in a similar manner for the constructed answer , but relative to the answer set and the correct answer of the task:
The input batches are shufled, to alternate between tasks during training, and avoid getting stuck in a local maximum for one of the tasks.
xxl-cased-discriminator. Multi-lingual Electra (E-M) model: google/electra-base-discriminator. 3To simplify the discussion of the method, we write "sentence" instead of "sentence embedding", when discussing the system.
Previous published work from our group and current
ongoing work has benchmarked the problems generated
by some of these datasets [
syntactic and semantic regularities –such as constituents, grammatical number and semantic roles– cannot be encoded together as they compete with each other when the system learns to distil them from the pretrained sentence embeddings.
Agr type_I-SingleTask type_I-Multitask
Caus
Task type_I -SingleTask type_I -Multitask type_I I-SingleTask type_I I-Multitask
Od
Figure 4: Performance comparison across single-task and Error Analysis For the agreement task, errors on the
multi-task training paradigms for the three subtasks (single grammatical number of the attractor nouns (WN1, WN2)
task darker shade of each colour, multi-task lighter shade), are high under both paradigms. These are "sequence
ertrained on type-I data, tested on the three types, and aver- rors", indicating that the system was not able to detect
aged over three independent runs. Results obtained using the the patterns in the input sequence, possibly because
inItalian Electra pretrained model. dividual sentence structures were not properly detected.
Previous experiments have shown, though, that in the
Discussion We expect that if the multi-task setup suc- single-task setting, the sentence level does manage to
ceeds in sharing information across tasks, then the re- compress the desired information [
As the results in Figure 4 show (and also the detailed re- and multi-task environments. We observe an overall insults in Tables 1-2 for the Italian Electra pretrained model, crease of error proportions in the multi-task environment. and in Tables 3-4 for a multilingual Electra pretrained Specifically, mistakes of the type I-int are frequent in model), single-task training outperforms multi-tasking type III data for the Caus task. These errors incorrectly in the agreement and verb alternation subtasks. The map the thematic roles onto the syntax of the arguments drop suggests that the multi-task model is not able to (e.g. L’artista si è chiuso ‘the artist closed’ or La carlearn shared properties for these tasks, and forcing it to bonara mangiava ‘the carbonara was eating’). In the do so leads to a model that is not optimal for either of same dataset, we also note an increase of errors related them. Both tasks require information about the syntactic to the last constituent in type I and type II data (errors structure (or sequence of phrases), while each requires of type E-WrBy, e.g. La finestra si chiuse dall’artista ‘the diferent phrase properties – grammatical number for window closed by the artist’). Finally, for the Od task, the agreement task, and semantic properties for the verb we remark that R-trans errors are not the most promialternation. While the system is able to distil all this in- nent —these are the errors resulting in standard transiformation from sentence embeddings in the single-task tive clauses (e.g., L’artista dipinse un paesaggio ‘the artist setting, it is not able to compress it into a shared repre- painted a landscape’)— and do not increase in multi-task sentation when learning the tasks together. environments, suggesting that the chosen answer is not
The Od single-task and multi-task have comparable derived from some forms of transitive bias [22]. performance, probably because the Od tasks involve a An overall comparison shows that the error patterns simpler alternation than the Caus task. They do not have vary across subtasks. This variety in error patterns cona causative meaning and do not require a change in the ifrms that the diferent dimensions (types of alternations, semantic role of the subjects. levels of lexicalisation and single and multi-task learning)
datasets of Italian on two linguistic phenomena of an heterogeneous nature, such as agreement and verbal transitive/intransitive alternation, embedded in the BLM task.
The results on the performance and the error analysis of a tailored two-level architecture have shown that multitask environments do not help, suggesting that abstract linguistic notions, such as constituents or thematic roles do not seem to be present in the learning process.
Current work is developing new analyses and architectures to probe further in the encoding of information in sentence embeddings and creating new BLM problems across various languages and linguistic phenomena.
0.12 0.10 ces Test., Psychological Review 97 (1990) 404–431. -main.75.
doi:10.1037/0033-295X.97.3.404. [17] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: [9] Z. Zhang, W. Yu, M. Yu, Z. Guo, M. Jiang, A sur- Pre-training of deep bidirectional transformers for vey of multi-task learning in natural language pro- language understanding, in: Proceedings of the cessing: Regarding task relatedness and training 2019 Conference of the North American Chapter of methods, in: A. Vlachos, I. Augenstein (Eds.), Pro- the Association for Computational Linguistics: Huceedings of the 17th Conference of the European man Language Technologies, Volume 1 (Long and Chapter of the Association for Computational Lin- Short Papers), Association for Computational Linguistics, Association for Computational Linguis- guistics, Minneapolis, Minnesota, 2019, pp. 4171– tics, Dubrovnik, Croatia, 2023, pp. 943–956. URL: 4186. URL: https://aclanthology.org/N19- 1423. https://aclanthology.org/2023.eacl- main.66. doi:10.18653/v1/N19-1423.
doi:10.18653/v1/2023.eacl-main.66. [18] K. Clark, M.-T. Luong, Q. V. Le, C. D. Manning, Elec[10] S. Chen, Y. Zhang, Q. Yang, Multi-task learning in tra: Pre- training text encoders as discriminators natural language processing: An overview, ACM rather than generators, in: ICLR, 2020, pp. 1–18.
Computing Surveys (2021). [19] B. Levin, English Verb Classes and Alternations A [11] S. Ruder, An overview of multi-task learn- Preliminary Investigation, University of Chicago ing in deep neural networks, arXiv preprint Press, Chicago and London, 1993. arXiv:1706.05098 (2017). [20] P. Merlo, S. Stevenson, Automatic verb classifica[12] J. Pfeifer, S. Ruder, I. Vulić, E. M. Ponti, Modu- tion based on statistical distributions of argument lar deep learning, arXiv preprint arXiv:2302.11529 structure, Computational Linguistics 27 (2001) 373– (2023). 408. [13] T. Standley, A. Zamir, D. Chen, L. Guibas, J. Malik, [21] P. A. Carpenter, M. A. Just, P. Shell, What one S. Savarese, Which tasks should be learned together intelligence test measures: a theoretical account of in multi-task learning?, in: International conference the processing in the raven progressive matrices on machine learning, PMLR, 2020, pp. 9120–9132. test., Psychological review 97 (1990) 404. [14] B. Zhou, X. Cai, Y. Zhang, X. Yuan, An end-to-end [22] K. Kann, A. Warstadt, A. Williams, S. R. Bowman, progressive multi-task learning framework for med- Verb argument structure alternations in word and ical named entity recognition and normalization, sentence embeddings, in: Proceedings of the Soin: C. Zong, F. Xia, W. Li, R. Navigli (Eds.), Proceed- ciety for Computation in Linguistics (SCiL) 2019, ings of the 59th Annual Meeting of the Association 2019, pp. 287–297. URL: https://aclanthology.org for Computational Linguistics and the 11th Interna- /W19-0129. doi:10.7275/q5js-4y86. tional Joint Conference on Natural Language Processing (Volume 1: Long Papers), Association for Computational Linguistics, Online, 2021, pp. 6214– 6224. URL: https://aclanthology.org/2021.acl-long.
485. doi:10.18653/v1/2021.acl-long.485. [15] Z. Hu, H. P. Chan, L. Huang, MOCHA: A multi-task training approach for coherent text generation from cognitive perspective, in: Y. Goldberg, Z. Kozareva, Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022, pp. 10324–10334. URL: https://aclanthology.org/2022. emnlp-main.705. doi:10.18653/v1/2022.emnlp -main.705. [16] Y. Wang, C. Zhai, H. Hassan, Multi-task learning for multilingual neural machine translation, in: B. Webber, T. Cohn, Y. He, Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 1022–1034. URL: https://aclanthology.org/2020.
emnlp-main.75. doi:10.18653/v1/2020.emnlp
A.1. An Italian example for the subject-verb agreement BLM
A.2. Verb alternation examples 1 2
Od, typeI - Context La turista mangia una carbonara in un secondo La turista mangia una carbonara da mezz’ora Una carbonara è mangiata dalla turista in un secondo Una carbonara è mangiata dalla turista da mezz’ora Una carbonara è mangiata in un secondo Una carbonara è mangiata da mezz’ora La turista mangia in un secondo ???
La zia mangia una bistecca nella sala grande La presidente può mangiare una bistecca da programma La specialità della casa deve essere mangiata dalla turista nella sala grande Una bistecca fu mangiata dalla presidente da sola La specialità della casa deve essere mangiata in un secondo Una bistecca deve poter essere mangiata da sola La turista deve mangiare con fame ???
L’attore deve canticchiare un motivetto dopo il festival L’amica di mia mamma deve cucire la tasca da qualche giorno L’inno nazionale può essere cantato dal vincitore del festival con solo pianoforte Una bistecca deve essere mangiata dalla turista da sola Il manuale è insegnato nell’aula magna Questi attrezzi devono essere intagliati da manuale I due fratelli studiano con molta attenzione ???
B.1. Results with the Italian Electra pretrained model: dbmdz/electra-base- italian-xxl-cased-discriminator B.2. Results with the multilingual Electra pretrained model: google/electra-base-discriminator