Large Language Models (LLMs) excel at understanding and generating text that appears human-written. Thus, it is intriguing to determine whether the models' text comprehension aligns in some way with human cognitive processes. A peculiarity of natural languages is that the same meaning can be encoded by multiple syntactic constructions. In Italian, for instance, the unmarked sentence follows a subject-verb-object (SVO) word order. However, inversions of this ordering do not necessarily lead to ungrammatical sentences. A case in point is represented by cleft sentence, i.e., sentences where the unmarked SVO sequence is violated. This corresponds to specific communicative functions, namely emphasize a component, and it is obtained by putting one element in a separate clause. In particular, Object Relative Clauseswhere the element that is emphasized is the object of the sentence -are difficult to understand [1,2]. For example the sentence "Sono i professori che i presidi hanno elogiato alla riunione d'istituto" is more challenging for an Italian speaker than its semantically equivalent unmarked version "I presidi hanno elogiato i professori alla riunione d'istituto" where the SVO order is restored. Similarly, in Nominal Copular constructions, the inversion of subject and verb clause is documented to cause difficulties in understanding the meaning of the sentence [3].
Hence, syntax plays a crucial role not only in the general construction of language but also in the native speakers ability to comprehend sentences: in fact, a correct syntactic parsing of the sentences is necessary to understand their meaning, and some syntactic structures are preferred over others. To what extent this preference is replicated by LLMs needs to be further explored.
If the model shows some knowledge about syntax, there should be an area of the model responsible for that. We aim to detect the area of a model responsible for its syntactic knowledge. Extensive work has been devoted to understanding how Transformer-based architectures encode information and one main objective is to localize which area of the model is responsible for a certain behavior [4,5]. Despite its usage as an explanation mechanism being debated [6,7], the attention mechanism is an interesting starting point given its wide use in Transformer architecture. While the attention weights alone cannot be used as an explanation of a model's behavior [8,9], an analysis that includes multiple components of the attention module is shown to be beneficial to obtain an interpretation of how a model processes an input sentence [10,11].
Probing is a common method used to detect the presence of linguistic properties of language in models [12]. Probing consists of training an auxiliary classifier on top of a model's internal representation, which could be the output of a specific layer, to determine which linguistic property the model has learned and encoded. In particular, it has been proposed to probe Transformerbased models to reconstruct syntactic representations like dependency parse trees from their hidden states [13]. Probing tasks concluded that syntactic features are encoded in the middle layers [14]. Correlation analysis on the weights matrices of the monolingual BERT models confirmed the localization of syntactic information in the middle layers showing that the models trained on syntactically similar languages were similar on middle layers [15]. While an altered word order seems to play a crucial role in Transformer-based models' ability to process language [16,17], the correlation between LLMs downstream performance and the encoding of syntax needs to be further explored.
In this paper, we initially examine how syntax influences the LLMs' capability of understanding language. To achieve this, we will analyze five open weights LLMs -trained on the Italian Language either from scratch or during a finetuning phase -and measure their performance in question-answering (Q&A) tasks that require an implicit parsing of the roles of words in the sentence to provide the correct answer. We use an available set of Q&A tasks designed for Italian speakers [1] and propose similar template-based questions for two other datasets of Italian sentences characterized by different syntactic structures (Section 2.1). The results show that the models are affected by the different syntactic structures in solving the proposed tasks (Section 3.1): LLMs struggle when more complex syntactic structures are present, with an average drop in accuracy of 16.13(Β±11.14) points.
We then propose a method -based on norm-based attribution [10]-to localize where syntactic knowledge is encoded by identifying the models' attention heads and layers that are responsible for the generation of a correct answer (Section 2.2). Although some differences can be observed across the five LLMs, we notice that syntactic information is more widely included in the middle and top layers of the models.
In this Section, we introduce the dataset we collected -largely extracted from the AcCompl-It task [18] in EVALITA 2020 [19] -to assess LLMs syntactic abilities. The dataset is split in three subdatasets. Each of the subdataset is composed of pairs of sentences that share the same meaning but a different word order. One of the sentences in each pair is characterized by a simpler structure, easier to understand also for humans, while the second is characterized by an alternative -but still correctsyntactic structure. We aim to understand whether a different structure can influence the model performance in processing those similar sentences. We define, for each subdataset, a Q&A task to assess the LLMs capabilities in understanding sentences when their syntactic structure makes them more complex. The Q&A task requires the model to implicitly parse the role of the words in the sentence to get the correct answer: for this reason, we identify some important words that the model should attend to while getting the correct answer.
The first subset is derived from Chesi and Canal [1]: this dataset contains 128 sentences characterized by Object Clefts (OC) constructions. The OC sentences in this dataset all share the same structure (see Table 1): the object and subject are words indicating either a person or a group of people, the predicate describes an action that the subject performs towards the object. The object is always introduced as the first element of the sentence in a left-peripheral position. The displacement of the object in the left-peripheral position makes the OC harder to understand [2]. We will compare those sentences with semantically equivalent ones that preserve the unmarked SVO word order.
To assess whether the difficulty humans have in understanding Object Cleft sentences can also be registered in LLMs for the Italian language, we tested them on the same Q&A task that Chesi and Canal [1] proposed to human subjects. Given one OC sentence, the model is prompted with a yes or no question asking whether one of the participants (subject or object) was involved in the action described by the predicate (see Table 1 for an example). The ability of a model to comprehend cleft sentences can be measured as the accuracy it obtains on this Q&A task. Moreover, we perform the same Q&A task on SVO sentences that we directly derived from the OC clauses in Chesi and Canal [1]: in this case, we restored the SVO order and produced sentences that are semantically equivalent to the corresponding OC (see Table 1).
To correctly solve the task, the model must interpret the role of the nouns of the sentences playing the role of subject and object to answer the comprehension question. Hence, the model should implicitly parse the sentences and focus on those relevant words during the generation of the answer.
The second subdataset -which includes 64 pairs of sentences-is derived from a study involving Nominal Copular constructions (NC) from Greco et al. [20]. The NC sentences are composed of two main constituents: a Determiner Phrase (π·π π π’ππ ) and a Verbal Phrase (π π ). The verbal phrase contains a copula and another Determiner Phrase that acts as the nominal part of the predicate (π·π ππππ ). In this dataset, the effect of the position of the subject with respect to the copular predicate is studied. Two semantically equivalent Table 1 Examples from the dataset under investigation. For each subdataset, an example is composed of two semantically equivalent sentences, that differ from the syntactic point of view, and a comprehension question on them.
sentences are presented for each example. In one case, the sentence presents a canonical structure (NC canonical), with the subject (π·π π π’ππ ) preceding the copular predicate. In the second case, an inverse structure (NC inverse) -with the subject following the predicate and the π·π ππππ introduced as the first element of the sentence -is presented (see Table 1). NC inverse sentences are syntactically correct but are harder to understand for humans than the NC canonical [3].
The structure of the sentences in this dataset is enriched by two Prepositional Phrases, one in each of the Determiner Phrases. The π·π π π’ππ includes a subject accompanied by an article and augmented with a Prepositional Phrase (π π π π’ππ ) that features a complement referring to the subject. Similarly, the π·π ππππ consists not only of a noun and an article but is instead further enriched with another Prepositional Phrase π π ππππ . The π π ππππ gives more information about the relation between the subject noun and the nominal part of the predicate.
We exploit the different role of the two Prepositional Phrases to design a Q&A task on NC canonical and NC inverse sentences and hence assess whether a more complex syntactic structure can influence LLMs capabilities. Given an NC sentence, the model is asked to correctly interpret the meaning of the sentence by examining its predicate: in particular, the model is asked to predict the additional information related to the nominal predicate -which is included in the π π ππππ -by answering a "wh-" question (in Italian, "Di cosa", see the example in Table 1). While both Prepositional Phrase answer to a wh-question, only the π π ππππ is related to the predicate of the sentence and hence the model should be able to predict the π π ππππ and ignore the π π π π’ππ .
To solve the proposed task and to properly understand NC sentences, humans and LLMs are required to im-plicitly parse the sentence and accurately identify the nominal part of the verbal phrase and, in particular, the Prepositional Phrase that it contains (π π ππππ ).
Finally, the last subdataset we investigate is derived from Greco et al. [20] and contains sentences characterized by minimal verbal structure (MVP). MVP sentences are composed of a subject, a predicate andfor sentences with transitive predicates -of an object (see Table 1). In this subdataset, the inversion of the subject and the verb is studied: the pairs of sentences under investigation have the same meaning (and lexicon) but in one cases the subject of the sentence follows the predicate (MVP post) while in the others the subject precedes the predicate (MVP pre). The latter configuration, in Italian, is more common that the former: we aim to investigate whether this syntactic variation can alter the performance of an LLM.
We define, for each pair of sentences, a question that asks the model to predict which element of the sentence is involved in a certain action, either as the subject entity or the object. In particular, for sentences that contain intransitive verbs, the model is always asked to predict the subject of the sentence, while in transitive cases (like the one in Table 1) the model is either asked to predict the subject or the object of the sentence. For this subdataset, while the original data included both declarative and interrogative sentences, we retained only the declarative ones: we test the model with a total of 192 sentence pairs.
To answer those questions, the relevant words -both for humans and LLMs -are the nouns that play the role of subject, or object if present, in sentences. In the next Section, we describe how it is possible to quantify whether a model is able to identify the role of those words during the generation of the answer.
Qwen2-7B LLaMAntino-3-ANITA-8B Llama-2-7b modello-italia-9b Meta-Llama-
Models accuracy on the different subdataset on the proposed Q&A tasks. Models tend to produce less accurate answers when exposed to more rare syntactic structures.
Knowing which sentence structures are easier or more difficult for a model to analyze is not enough. Considering the black-box nature of these models, it is essential to understand which layers are responsible for encoding syntax, thus making the models more interpretable. We hypothesize that there is an area of the model responsible for correctly analyzing the sentence from the syntactic point of view in order to get the answer to the Q&A task. In fact, as discussed in the previous Section, to answer correctly, the model needs to implicitly parse the roles of the words in the sentence and identify the relevant words for the response (subjects and objects in the questions on OC, SVO and MVP sentences and the correct prepositional phrases in NC sentences). Hence, a knowledge of syntax is required to identify the relevant words and, consequentially, generate the correct answer.
In generating the answer, we expect the model to "focus" on those relevant words. We can identify to which token the model focuses during generation, measuring token-to-token attributions [8,10]. In fact, token-totoken attribution methods quantify the influence of a token in the generation of the other. We argue that the part of the model architecture most aware of syntax is the one that systematically focuses on relevant words when the model is prompted to answer syntax-related questions. Kobayashi et al. [10] demonstrate that a mechanism -called the norm-based attribution -that it incorporates also the dense output layer of the Attention Mechanism is an accurate metric for token-to-token attribution. We will refer to the matrix π΄ β (π) -computed for the attention head β for a sequence π -as an attribution matrix. Some examples and a more detailed description of norm-based attribution can be found in the Appendix (A.1). The attribution matrix π΄ β (π), for each sequence of tokens π, describes where the model focuses during the generation of each token. By examining all the attention heads, some of them may focus more often on the subject, the object, or the prepositional phrase in the predicate while generating the answer for the task. In particular, for each attention head β, we consider the tokens to be attributed for the generated answer produced by the model: for each correct answer generated by the model, we count the number of times the tokens with the larger attribution value are the relevant ones. This measures the accuracy of the attention head β in recognizing the relevant words to generate the answer.
The more often the attention head focuses on the relevant words, the more syntactic knowledge the head encodes. For each downstream task presented in Section 2.1, we collect the accuracy of all heads at all levels. Then, we identify a head as "responsible" for generating the target word in a task if its score is higher than the average score for that task. Specifically, we assume a Gaussian distribution of scores for each task and identify a head as responsible if the probability of observing a value at least as extreme as the one observed is below a threshold πΌ < 0.05. We also consider responsible all heads that obtain an excellent accuracy score (greater than 0.9) in focusing on the relevant words. With this procedure, for each layer and task, we can localize the responsible heads and determine where the model encodes syntax the most.
We focus on Instruction-tuned LLMs, all of comparable size, and trained -either from scratch or only fine-tuned -on the Italian language. The models1 under investigation are Qwen2-7B [22], LLaMAntino-3-ANITA-8B [23], Llama-2-7b [24], modello-italia [25], and Meta-Llama-3-8B [26]. To solve the Q&A task, we prompted each model with 4 different -but semantically equivalentinstructions. The complete list of the prompts is in Appendix A.2. All prompts ask the model to solve the task in zero-shot by answering only with one or two words. At most 128 tokens are generated, with greedy decoding. Once the generation is completed, a manual check of the responses is performed to obtain a simplified response to be compared with the gold. For the subsequent analysis, for each model and task, only the prompt for which the higher accuracy is obtained is considered.
We initially revise model's accuracy on question comprehension task and assess models capabilities when different syntactic structures are involved (Section 3.1). Then, we aim to spot the layers responsible for the correct syntactic understanding of the sentences (Section 3.2).
Results on each of the subdatasets show that the syntactic structure of a sentence influences the models' understanding of that sentence (see Table 2): across all tasks, LLMs tend to obtain larger accuracy on sentences characterized by a unmarked syntactic structure.
On the first task, on OC and SVO sentences, the models tend to struggle, especially in the OC sentences. On OC sentences, some models, in fact, do not perform far from the random baseline of 50% accuracy ("yes" and "no" answers are balanced). When comparing OC and SVO sentences, on average, the model accuracy drops by 11.88(Β±3.84) points when the sentence presents the object in the left-peripheral position. This result aligns with the difficulty that humans encounter in understanding those sentences. The model that achieves the highest accuracy in this task in OC sentences is LLaMAntino-3-ANITA-8B, with an accuracy of 76.56. It is important to note that the model performance increase of 11.72 points with respect to the corresponding Meta-LLama-3-8b (that achieves an accuracy of 64.84): these results stress the effectiveness of the finetuning for the Italian language. Across the LLaMa-based models the LLaMAntino-3-ANITA-8B is still the best performing model, followed by Meta-LLama-3-8b and with a larger gap by LLama-2-7b. The Qwen2-7B model is the best answering to the task on unmarked sentences.
On the NC sentences, similar patterns to the one ob-served in the previous subdataset emerge. In particular, the NC inverse sentences are harder than the corresponding NC canonical: the average model accuracy is 81.88(Β±11.78) on NC canonical sentences, while the accuracy on NC inverse sentences is much lower, with an average value of 64.06(Β±28.26). Also in this case, the results demonstrate that models are affected by different syntactic patterns. The model that better capture the right information to extract is modello-italia-9b on both NC inverse and NC-canonical sentences. Although the performance of Llama-2-7b is rather low on inverse NC sentences (the model tends to generate very often the π π π π’ππ ), the remaining LLaMA-base models achieve better performance on both tasks. Finally, results on the MVP task further confirm the models' behavior observed on the previous two tasks: the inversion of the subject and verb positions causes the models to perform worst on MVP post sentences (87.5(Β±19.38) average accuracy) with respect to MVP pre (68.23(Β±10.37) average accuracy). The average drop in performance is larger than in previous subtasks: these results confirm that the inversion of the subject, even in basic sentences, can degrade models' understanding. Modello-italia-9b -probably due to the limited length of the input sentences -tends to replicate the input sentences. The other models solve the tasks with excellent accuracy in the MVP pre sentences.
After quantifying the impact of different syntactic structures on model performance, we can identify the attention heads and levels of the models that mostly encodes syntax. In Figure 1 the number of responsible head at each layer of the models is reported for the Q&A task on NC sentences, (the remaining tasks are in Appendix A.3). The general trend is that the most active in identifying relevant words during response generation layers are comprised between layer 19 and 25. Moreover, for all models, the layers we identify as responsible often handle multiple syntactic structures. The most noticeable result is that for the same task, the same activation trend emerges across all sentences.
A large number of responsible attention heads appear around layer 19 to 27 in LLaMAntino-3-ANITA-8B and Meta-Llama-3-8B. Layer 21, in particular, is the layer with the most responsible heads both in NC and MVP tasks. This layer is predominant also in the OC task, concomitant with layers 19 and 22 (Figure 3a). For Llama-2, we observe the same pattern as the most active layers are between 18 and 25. On the Qwen2-7B model and modelloitalia-9b active layers are higher in the architecture: from layer 18 to 24 for Qwen2-7B (with layer 23 being the more active in NC and MVP tasks) and from layer 21 to 31 on NC and MVP senteces for modello-italia-9b. This finding suggests a different interpretation of LLMs layers from that previously observed in BERT [27].
While we could expect some correlation between the accuracy of the task and the capability of the model to identify the correct word in the sentence, the responsible heads appear to be shared across different syntactic structures. Those results suggest that some layers, more than others, encode syntactic information about the role of a word in a sentence. Moreover, different models and architectures seem to share a rather similar organization.
In this paper, we have investigated how semantically equivalent sentences are processed by LLMs in Italian when their syntax differs. We tested LLMs trained on the Italian -or with Italian data in the pre-trainig material -and measured how their capabilities in a battery of Q&A tasks that rely on parsing the correct role of words in a sentence to be solved. Our findings confirm that cleft sentences and construction with an inversion of subject and verb are difficult to understand also for LLMs -similarly to what observed for humans. Furthermore, we have identified systematically using token-to-token attribution that syntactic information tends to be encoded in the middle and top layers of LLMs.
As described in Section 2.2, we adopt norm-based token-to-token attribution to spot what is the most relevant word during the generation of the answer in LLMs on our task. The norm based approach is proposed in Kobayashi et al. [10]. Given the query weight matrix π β π , key weight matrix π β πΎ , value weight matrix ππ and the attention output weight matrix π β π of an attention head β, the norm-based attribution for each token of a sequence π is calculated as the product of the attention weights and the norm of the projected token representation ππ β π π β π (see the original work Kobayashi et al. [10] for a detailed discussion).
)οΈ
β’ βππ β π π β π β For our analysis, we consider all rows relative to a token in the answer generated by the model. To assess whether a model understands the syntactic relationship between words, it must focus on relevant words during the generation. In particular, the token with the highest attribution should be one belonging to the relevant word. For example, in Figure 2, the attribution of Meta-Llama-3-8B on one NC sentence is presented. During the generation of the answer (the tokens of the answer index rows in the figure), the most attributed tokens belong to the relevant words in the input (the tokens of the input index columns).
Each model has been prompted with four different prompts for each Q&A task (as described in Section 2.1).
Here is a complete list of the prompts template used in our experiments: in the template the {Item} is the sentence to be analyzed and {Question} is replaced with the corresponding comprehension question. OC and SVO senteces:
β’ Data la frase "{Item}", rispondi alla seguente domanda:"{Question}" Rispondi SOLAMENTE con SI o NO. β’ Considera la frase: "{Item}". Rispondi con 'SI' o 'NO' alla seguente domanda:"{Question}" β’ Considera la frase: "{Item}". {Question} Rispondi brevemente, SOLAMENTE con con 'SI' o 'NO'. β’ Considera la frase: "{Item}". Rispondi con 'SI' o 'NO'. {Question} NC sentences:
β’ Data la frase "{Item}", rispondi alla seguente domanda:"{Question}" Rispondi in due parole. β’ Considera la frase: "{Item}". Rispondi solo con le due parole che rispondono alla seguente domanda:"{Question}" β’ Considera la frase: "{Item}". {Question} Rispondi SOLO con le due parole che rispondono alla seguente domanda. β’ Considera la frase: "{Item}". Rispondi solo con due parole. {Question} MVP sentences:
β’ Data la frase "{Item}", rispondi alla seguente domanda:"{Question}" Rispondi solo con un nome. β’ Considera la frase: "{Item}". Rispondi solo con il nome che risponde alla seguente domanda:"{Question}" β’ Considera la frase: "{Item}". {Question} Rispondi SOLO con il nome che risponde alla domanda. β’ Considera la frase: "{Item}". Rispondi solo con un nome. {Question}
In Figure 3, the responsible attention heads per layer is depicted. As described in Section 3.2, some layers tend to demonstrate a high number of attention heads responsible for the generation. In particular, layers around layer 20 seem to focus more on relevant words for the correct generation of the answer than the other. Since the correct generation implies the capability of understanding the role of different words by a model, we claim that those level encodes some kind of syntactic information.
It is worth noticing that similar layers are responsible for the different sub tasks, in particular for the LLaMa-base models and for Qwen-2-7b model.