Driven by increasing computational scale and progress in deep learning techniques, NLP models can rival human capabilities on established benchmarks. New benchmarks, then, that capture deeper levels of language understanding must be created and analysed [1].
Blackbird's Language Matrices (BLM) [2] is a recent task inspired by visual tests of analytic intelligence (Raven Progressive Matrices/RPMs, [3]). The BLM tasks have cast light on whether the correct predictions in previously studied linguistic problems, e.g. number agreement or verb alternations, stem from sentence embeddings that encode deeper linguistic information, such as syntactic structure and semantic properties of phrases [4,5,6]. We found that higher-level information -syntac-tic structure and argument structure -can be assembled from the information encoded in the sentence embeddings. This, however, may not be due to a deeper understanding of such information encoded by LLMs, but rather because of useful surface indicators [7].
In this paper, we adopt BLMs to investigate whether current pretrained models encode abstract linguistic notions, such as constituents, and are able to do so in a manner that comprises both functional elements, such as pronouns, demonstratives and lexical elements, such as nominal constituents.
We concentrate on Italian, and study several grammatical problems whose solutions can theoretically help each other, in a multi-task setting. We adopt a two-level architecture developed specifically to model what we know about how humans solve puzzles similar to BLMs [8]. Level 1 aims to obtain compressed sentence representations that capture information about constituents and their properties; level 2 uses the compressed sentence representations to solve a BLM problem. This architecture provides a tool to study how LLMs encode different types of syntactic and semantic information.
We make two contributions: (i) an initial core BLM dataset for Italian that covers linguistic problems of different nature; (ii) single and multi-task experiments that provide new insights into the information encoded by LLMs. The datasets are available at https://www.idiap.ch/datas et/(blm-agri|blm-causi|blm-odi) and the code at https: //github.com/CLCL-Geneva/BLM-SNFDisentangling.
Multi-task learning has been popular in improving NLP systems' performance by using knowledge shared across multiple tasks [9].
Multi-task learning architectures include parallel, hierarchical, and modular designs [10]. Parallel architectures share intermediate layers across tasks, conducive to efficient knowledge transfer [11]. Hierarchical architectures capture task dependencies by layering task-specific modules on shared bases. Modular approaches selectively share components among tasks to balance between generalisation and task-specific optimisation [12]. These training strategies are not mutually exclusive and can be combined.
Multi-task learning can be used efficiently in resourceconstrained environments, to counter data scarcity and overfitting: aggregating training data and sharing parameters across related tasks acts as a form of data augmentation [13].
Effective multi-task learning depends on the relatedness of the tasks involved. Tasks that are similar or have similar objectives tend to benefit more from shared representations. This observation has been used in various NLP tasks, including named entity recognition [14], text generation [15], and machine translation [16], among others. Selecting related tasks that contribute positively to the shared model's training is important and remains an active area of research [9].
Pretrained large language models exhibit generalpurpose abilities and knowledge, with high results with little or no fine-tuning on downstream tasks [17,18]. We can then regard these language models as the results of "multi-task" learning, and our aim here is to test whether sentence embeddings obtained from these models encode syntactic and semantic information consistently, such that different BLM problems that rely on similar linguistic information draw on the same clues from these representations. In particular, we will use BLM tasks on subject-verb agreement -which relies on chunk structure and the chunks' grammatical number propertiesand on verb alternations -which relies on chunk structure and the chunks' semantic role properties -to test whether chunk structure is encoded in a manner that allows for it to be shared by the two tasks.
Raven's progressive matrices are multiple-choice completion IQ tests, whose solution requires discovering underlying generative rules of a sequence of images [3].
A similar task has been developed for linguistic problems, called Blackbird Language Matrices (BLMs) [2], as given in Figure 1, which illustrates the template of a BLM agreement matrix. A BLM comprises a context and an answer set. The context is a sequence of sentences generated following the relevant rules of a given linguistic phenomenon under investigation and that this way implicitly illustrates these grammatical properties. This sequence also follows some extra-linguistic progression rules. Each context is paired with a set of candidate answers. The answer sets contain minimally contrastive examples built by corrupting some of the generating rules.
The BLM Italian datasets consists of BLMs focused on the property of subject-verb agreement and two transitive-intransitive alternations: the change-of-state alternation and the object-drop alternation.
The BLM-AgrI dataset is created by manually translating the seed French sentences [4] into Italian by a native speaker, one of the authors, and then generating the full dataset following the same process of lexical augmentation and sentence shuffling among instances described in [4]. The internal nominal structure in these languages is very similar, so translations are almost parallel. An illustrative, simplified example for Italian is provided in Figure 7, in the appendix. The dataset comprises three subsets of increasing lexical complexity (called Type I, Type II and Type III) to test the ability of the system to handle item novelty.
While BLM-AgrI tests information about a formal grammatical property, agreement, the Causative (Caus) and Object-drop (Od) alternation datasets test lexical semantic properties of verbs, their ability to enter or not a causative alternation. Caus represents the causative/inchoative alternation, where the object of the transitive verb bears the same semantic role (Patient) as the subject of the intransitive verb (L'artista ha aperto la finestra/La finestra si Γ¨ aperta 'The artist opened the window'/'The window opened'). The transitive form of the verb has a causative meaning. In contrast, the subject in Od bears the same semantic role (Agent) in both the transitive and intransitive forms (L'artista dipingeva la finestra/L'artista dipingeva 'the artist painted the window'/'the artist painted') and the verb does not have a causative meaning [19,20].
The context set of the verb alternation varies depending on the presence of one or two arguments and their attributes (agents, Ag; patients, Pat) and the active (Akt) and passive (Pass) or passive voice of the verb. The non-linguistic factor that structures the sequence is an alternation every two items between a prepositional phrase introduced by any preposition (e.g., in pochi secondi, P-NP) and a PP introduced by the agentive da-NP (e.g., dall'artista, da-Ag/da-Pat).
The answer set is composed of one correct answer and contrastive wrong answers, all formed by the same four elements: a verb, two nominal constituents and a prepositional phrase. Figure 2 shows the template.1
The BLM for Od is the same as for Caus, but here the passive voice serves as a confounding element and one of the contrastive answers for Caus is, in fact, the correct answer here.
The template is also in Figure 2. Due to the asymmetry between the two classes of verbs, the contexts of the BLMs minimally differ in the intransitive followed by P-NP (sentence 7). The correct answer also varies across the two groups, although in both cases it is an intransitive form with a da-NP. Examples are shown in the Appendix. We illustrate the data in Figure 8 in the appendix with the Italian Change-of-state verb chiudere 'close'.
In line with previous work on BLMs, each dataset also contains a varying amount of lexicalisation. In type I the lexical material of the sentences within a single context does not change, in type II only the verb remains the same, in type III data all words can change (Figure 9, in the appendix).
Each subset is split 90:20:10 into train:dev:test subsets. The training and testing are disjoint (agreement data is split based on the correct answer, the alternations data based on the verb). Agreement has 230 test instances for type I, 4121 for types II and III. The verb alternations have 240 test instances for all subsets. We randomly sample a number of training instances, depending on the experimental set-up.
Sentence embeddings encode much information from the input sentence -lexical, syntactic, semantic, and possibly other types of information. Previous experiments have shown that sentence embeddings can be compressed into very small representations (vectors of size 5) that encode information about the structure of the sentence in terms of chunks and their properties, such that they contribute to finding the sequence patterns in BLMs [6]. In this work, we investigate whether several BLM tasks can share the same structural information from a sentence embedding. Towards this end, we built a multi-task version of a two-level system, illustrated in Figure 3. In this system, one level processes individual sentences and learns to compress them into small vectors that retain information pertinent to a task and the other level uses the compressed sentence representation to find patterns across an input sequence to solve a BLM task. The multitask variation consists in a single shared sentence-level component, and multiple task components, one for each of the BLM tasks.
The BLM problems encode a linguistic phenomenon through data that has structure on multiple levelswithin sentences, and across a sequence of sentences. We can exploit this structure to develop an indirectly supervised approach to discover and use these different levels of structure. We thus model the solving of a BLM task as a two-step process: (i) compress individual sentences into a representation that emphasizes the sentence structure relevant to the BLM problem (e.g. chunks and their grammatical number for the subject-verb agreement task) (ii) use the compressed representations to detect the sequence-level pattern and solve the BLM task. This two-step process has been shown to be used by people solving visual intelligence tests [21]. In our case, this setup allows us to investigate whether the sentence level can be guided to learn shared information, relevant to the different linguistic tasks described in section 3.
We implement this approach in the two-level intertwined architecture illustrated in Figure 3, and described in detail elsewhere [6]. The data is pre-encoded with Electra [18]. 2 The sentence representations is provided by the embedding of the [CLS] token. 3 . We chose Electra because of its stronger sentence-level supervision signal, which leads to higher results when testing the encoding of structural information compared to BERT, RoBERTa, and models tuned by semantic similarity [6].
The two levels are learned together. The input is a BLM instance which is processed on the fly to produce training instances for the sentence level for each sentence ππ π in the input sequence π. The compressed sentence representations on the latent layer π§ππ π are stacked and passed as input to the task level, which produces a sentence representation πππ π€ as output, which is compared to the answer set of the respective BLM instance π΄.
The sentence level uses a variational encode-decoder architecture to learn how to compress on the latent layer a representation that captures relevant structural information. We guide the system towards this representation by constructing a contrastive set of candidates for comparison with the reconstructed input. The correct output (ππ’π‘ + ) is the same as the input (ππ), and a selection of other sentences from the input sequence will be the contrastive negative outputs (ππ’π‘ β = {ππ’π‘ β π , π = 1, πππππ }, πππππ = 7 (note that an input sequence consists of sentences with different patterns to each other -Figure 1 and 2). We use a max-margin loss function to take advantage of the contrastive answers, ππ Λis the reconstructed input sentence from the sampled latent vector π§ππ:
)
The loss at the task level for input sequence π is computed in a similar manner for the constructed answer πππ π€, but relative to the answer set π and the correct answer ππ of the task:
+ πΎπΏπ ππ(π§π|π© (0, 1)).
The loss of the two-level systems is:
The input batches are shuffled, to alternate between tasks during training, and avoid getting stuck in a local maximum for one of the tasks.
Previous published work from our group and current ongoing work has benchmarked the problems generated by some of these datasets [4,5]. This work has shown that information about the syntactic phrases in a sentence and their properties can be obtained from sentence embeddings, and this information is helpful in solving the BLM tasks. We had studied these tasks separately, and investigate here whether such structure is encoded in the sentence embeddings, or whether it is assembled based on shallower patterns within the sentence representations.
We expect that if the multi-task setup succeeds in sharing information across tasks, then the results on the individual test data will be at least as good as when learning tasks individually, given that the multitask setup uses a larger training set data -the union of the training sets of the individual tasks. But, overall, this does not seem to be the case. As the results in Figure 4 show (and also the detailed results in Tables 1-2 for the Italian Electra pretrained model, and in Tables 3-4 for a multilingual Electra pretrained model), single-task training outperforms multi-tasking in the agreement and verb alternation subtasks. The drop suggests that the multi-task model is not able to learn shared properties for these tasks, and forcing it to do so leads to a model that is not optimal for either of them. Both tasks require information about the syntactic structure (or sequence of phrases), while each requires different phrase properties -grammatical number for the agreement task, and semantic properties for the verb alternation. While the system is able to distil all this information from sentence embeddings in the single-task setting, it is not able to compress it into a shared representation when learning the tasks together.
The Od single-task and multi-task have comparable performance, probably because the Od tasks involve a simpler alternation than the Caus task. They do not have a causative meaning and do not require a change in the semantic role of the subjects. The comparison of all the tasks suggests that some syntactic and semantic regularities -such as constituents, grammatical number and semantic roles-cannot be encoded together as they compete with each other when the system learns to distil them from the pretrained sentence embeddings.
Error Analysis For the agreement task, errors on the grammatical number of the attractor nouns (WN1, WN2) are high under both paradigms. These are "sequence errors", indicating that the system was not able to detect the patterns in the input sequence, possibly because individual sentence structures were not properly detected. Previous experiments have shown, though, that in the single-task setting, the sentence level does manage to compress the desired information [6]. The fact that both these errors increase in the multi-task setting indicates that the information compression on the sentence level is less successful than in the single-task setting.
For the alternation tasks, error patterns vary, although their distributions remain similar between single-task and multi-task environments. We observe an overall increase of error proportions in the multi-task environment. Specifically, mistakes of the type I-int are frequent in type III data for the Caus task. These errors incorrectly map the thematic roles onto the syntax of the arguments (e.g. L'artista si Γ¨ chiuso 'the artist closed' or La carbonara mangiava 'the carbonara was eating'). In the same dataset, we also note an increase of errors related to the last constituent in type I and type II data (errors of type E-WrBy, e.g. La finestra si chiuse dall'artista 'the window closed by the artist'). Finally, for the Od task, we remark that R-trans errors are not the most prominent -these are the errors resulting in standard transitive clauses (e.g., L'artista dipinse un paesaggio 'the artist painted a landscape')-and do not increase in multi-task environments, suggesting that the chosen answer is not derived from some forms of transitive bias [22].
An overall comparison shows that the error patterns vary across subtasks. This variety in error patterns confirms that the different dimensions (types of alternations, levels of lexicalisation and single and multi-task learning) are separate uncorrelated dimensions. It also indicates that the differences in the F1 results shown in Figure 4 are real, despite the more homogeneous trends exhibited by these aggregated F1 numbers.
In this paper, we have presented curated synthetic datasets of Italian on two linguistic phenomena of an heterogeneous nature, such as agreement and verbal transitive/intransitive alternation, embedded in the BLM task.
The results on the performance and the error analysis of a tailored two-level architecture have shown that multitask environments do not help, suggesting that abstract linguistic notions, such as constituents or thematic roles do not seem to be present in the learning process.
Current work is developing new analyses and architectures to probe further in the encoding of information in sentence embeddings and creating new BLM problems across various languages and linguistic phenomena.
We gratefully acknowledge the partial support of this work by the Swiss National Science Foundation, through grant SNF Advanced grant TMAG-1_209426 to PM.
A.1. An Italian example for the subject-verb agreement BLM Context 1 Il vaso con il fiore si Γ¨ rotto. 2 I vasi con il fiore si sono rotti. 3 Il vaso con i fiori si Γ¨ rotto. 4 I vasi con i fiori si sono rotti. 5 Il vaso con il fiore del giardino si Γ¨ rotto. 6 I vasi con il fiore del giardino si sono rotti. 7 Il vaso con i fiori del giardino si Γ¨ rotto. 8 ???
Answer set 1 Il vaso con i fiori e il giardino si Γ¨ rotto. coord 2 I vasi con i fiori del giardino si sono rotti. correct 3 Il vaso con il fiore si Γ¨ rotto. WNA 4 I vasi con il fiore del giardino si sono rotti. WN1 5 I vasi con i fiori dei giardini si sono rotti. WN2 6 Il vaso con il fiore del giardino si sono rotti. AEV 7 Il vaso con i fiori del giardino si sono rotti. AEN1 8 Il vaso con il fiore dei giardini si sono rotti. AEN2
Caus -Context 1 Una stella del cinema chiuse la sua carriera con forza 2 Una stella del cinema chiuse la sua carriera da pochissimo tempo 3 La sua carriera fu chiusa da una stella del cinema con forza 4 La sua carriera fu chiusa da una stella del cinema da pochissimo tempo 5 La sua carriera fu chiusa con forza 6 La sua carriera fu chiusa da pochissimo tempo 7 La sua carriera si chiuse con forza 8 ???
Caus -Answers 1 La sua carriera si chiuse da pochissimo tempo 2 Una stella del cinema si chiuse da pochissimo tempo 3 La sua carriera fu chiusa da una stella del cinema 4 Una stella del cinema fu chiusa dalla sua carriera 5 La sua carriera chiuse una stella del cinema 6 Una stella del cinema chiuse la sua carriera 7 La sua carriera si chiuse da una stella del cinema 8 Una stella del cinema si chiuse dalla sua carriera Od, typeI -Context 1 La turista mangia una carbonara in un secondo 2 La turista mangia una carbonara da mezz'ora 3 Una carbonara Γ¨ mangiata dalla turista in un secondo 4 Una carbonara Γ¨ mangiata dalla turista da mezz'ora 5 Una carbonara Γ¨ mangiata in un secondo 6 Una carbonara Γ¨ mangiata da mezz'ora 7 La turista mangia in un secondo 8 ???
Od, typeI -Answers 1 Una carbonara mangia da mezz'ora 2 La turista mangia da mezz'ora 3 Una carbonara Γ¨ mangiata dalla turista 4 La turista Γ¨ mangiata da una carbonara 5 Una carbonara mangia la turista 6 La turista mangia una carbonara 7 Una carbonara mangia dalla turista 8 La turista mangia da una carbonara Od, typeII -Context 1 La zia mangia una bistecca nella sala grande 2 La presidente puΓ² mangiare una bistecca da programma 3 La specialitΓ della casa deve essere mangiata dalla turista nella sala grande 4 Una bistecca fu mangiata dalla presidente da sola 5 La specialitΓ della casa deve essere mangiata in un secondo 6 Una bistecca deve poter essere mangiata da sola 7 La turista deve mangiare con fame 8 ???
Od, typeII -Answers 1 La specialitΓ della casa puΓ² mangiare da sola 2 La squadra di calcio deve mangiare da mezz'ora 3 Una bistecca Γ¨ mangiata dalla turista 4 La squadra di calcio puΓ² essere mangiata da una carbonara 5 La pasta col pomodoro puΓ² mangiare la squadra di calcio 6 La squadra di calcio mangia una bistecca 7 La specialitΓ della casa deve poter mangiare dalla turista 8 La presidente mangia da una bistecca Od, typeIII -Context 1 L'attore deve canticchiare un motivetto dopo il festival 2 L'amica di mia mamma deve cucire la tasca da qualche giorno 3 L'inno nazionale puΓ² essere cantato dal vincitore del festival con solo pianoforte 4 Una bistecca deve essere mangiata dalla turista da sola 5 Il manuale Γ¨ insegnato nell'aula magna 6 Questi attrezzi devono essere intagliati da manuale 7 I due fratelli studiano con molta attenzione 8 ???
Od, typeIII -Answers 1 La pasta frolla deve impastare da sola 2 L'autrice deve poter scrivere da qualche giorno 3 I libri di testo devono poter essere studiati dai candidati 4 Questi stilisti devono poter essere tessuti dai vestiti per la parata 5 Questi motivi greci possono tessere questi stilisti 6 L'idraulico saldΓ² i cavi del lampadario 7 La stanza pulisce da una delle propretarie dell'albergo 8 Le sommozzatrici pescarono da delle trote