While LLMs get more proficient at solving tasks and generating sentences, we aim to investigate the role that diferent syntactic structures have on models' performances on a battery of Natural Language Understanding tasks. We analyze the performance of five LLMs on semantically equivalent sentences that are characterized by diferent syntactic structures. To correctly solve the tasks, a model is implicitly required to correctly parse the sentence. We found out that LLMs struggle when there are more complex syntactic structures, with an average drop of 16.13(± 11.14) points in accuracy on Q&A task. Additionally, we propose a method based on token attribution to spot which area of the LLMs encode syntactic knowledge, by identifying model heads and layers responsible for the generation of a correct answer.
Large Language Models (LLMs) excel at understanding and generating text that appears human-written. Thus, it is intriguing to determine whether the models’ text comprehension aligns in some way with human cognitive processes. A peculiarity of natural languages is that the same meaning can be encoded by multiple syntactic constructions. In Italian, for instance, the unmarked sentence follows a subject-verb-object (SVO) word order.
However, inversions of this ordering do not
necessarily lead to ungrammatical sentences. A case in point is
represented by cleft sentence, i.e., sentences where the
unmarked SVO sequence is violated. This corresponds to
specific communicative functions, namely emphasize a
component, and it is obtained by putting one element in
a separate clause. In particular, Object Relative Clauses –
where the element that is emphasized is the object of the
sentence – are dificult to understand [
Hence, syntax plays a crucial role not only in the general construction of language but also in the native speakers ability to comprehend sentences: in fact, a correct syntactic parsing of the sentences is necessary to understand their meaning, and some syntactic structures are preferred over others. To what extent this preference is replicated by LLMs needs to be further explored.
If the model shows some knowledge about syntax, there should be an area of the model responsible for that.
We aim to detect the area of a model responsible for its
syntactic knowledge. Extensive work has been devoted
to understanding how Transformer-based architectures
encode information and one main objective is to
localize which area of the model is responsible for a certain
behavior [
Probing is a common method used to detect the presence of linguistic properties of language in models [12].
Probing consists of training an auxiliary classifier on
top of a model’s internal representation, which could be
the output of a specific layer, to determine which
linguistic property the model has learned and encoded. In
particular, it has been proposed to probe
Transformerbased models to reconstruct syntactic representations
like dependency parse trees from their hidden states [13]. subdataset, a Q&A task to assess the LLMs capabilities in
Probing tasks concluded that syntactic features are en- understanding sentences when their syntactic structure
coded in the middle layers [14]. Correlation analysis on makes them more complex. The Q&A task requires the
the weights matrices of the monolingual BERT models model to implicitly parse the role of the words in the
confirmed the localization of syntactic information in sentence to get the correct answer: for this reason, we
the middle layers showing that the models trained on identify some important words that the model should
syntactically similar languages were similar on middle attend to while getting the correct answer.
layers [15]. While an altered word order seems to play
a crucial role in Transformer-based models’ ability to Object Clefts constructions The first subset is
deprocess language [16, 17], the correlation between LLMs rived from Chesi and Canal [
In this paper, we initially examine how syntax influ- same structure (see Table 1): the object and subject are
ences the LLMs’ capability of understanding language. words indicating either a person or a group of people, the
To achieve this, we will analyze five open weights LLMs predicate describes an action that the subject performs
– trained on the Italian Language either from scratch or towards the object. The object is always introduced as
during a finetuning phase – and measure their perfor- the first element of the sentence in a left-peripheral
posimance in question-answering (Q&A) tasks that require tion. The displacement of the object in the left-peripheral
an implicit parsing of the roles of words in the sentence position makes the OC harder to understand [
We then propose a method – based on norm-based the action described by the predicate (see Table 1 for an
attribution [
Hence, the model should implicitly parse the sentences LLMs Syntactic Abilities and focus on those relevant words during the generation of the answer.
– largely extracted from the AcCompl-It task [18] in
EVALITA 2020 [19] – to assess LLMs syntactic abilities. The Copular Constructions The second subdataset
The dataset is split in three subdatasets. Each of the sub- –which includes 64 pairs of sentences– is derived from
dataset is composed of pairs of sentences that share the a study involving Nominal Copular constructions (NC)
same meaning but a diferent word order. One of the sen- from Greco et al. [20]. The NC sentences are composed
tences in each pair is characterized by a simpler structure, of two main constituents: a Determiner Phrase ( )
easier to understand also for humans, while the second and a Verbal Phrase ( ). The verbal phrase contains a
is characterized by an alternative – but still correct – copula and another Determiner Phrase that acts as the
syntactic structure. We aim to understand whether a dif- nominal part of the predicate (). In this dataset,
ferent structure can influence the model performance in the efect of the position of the subject with respect to the
processing those similar sentences. We define, for each copular predicate is studied. Two semantically equivalent
SVO
Question
NC inverse
NC canonical
Question
MVP post
MVP pre
Question
sentences are presented for each example. In one case, plicitly parse the sentence and accurately identify the
the sentence presents a canonical structure (NC canon- nominal part of the verbal phrase and, in particular, the
ical), with the subject ( ) preceding the copular Prepositional Phrase that it contains ( ).
predicate. In the second case, an inverse structure (NC
inverse) –with the subject following the predicate and Minimal Verbal Structure with Inversion of Subject
the introduced as the first element of the sen- and Verb Finally, the last subdataset we investigate
tence – is presented (see Table 1). NC inverse sentences is derived from Greco et al. [20] and contains sentences
are syntactically correct but are harder to understand for characterized by minimal verbal structure (MVP). MVP
humans than the NC canonical [
The structure of the sentences in this dataset is en- for sentences with transitive predicates – of an object riched by two Prepositional Phrases, one in each of the (see Table 1). In this subdataset, the inversion of the Determiner Phrases. The includes a subject ac- subject and the verb is studied: the pairs of sentences companied by an article and augmented with a Preposi- under investigation have the same meaning (and lexicon) tional Phrase ( ) that features a complement refer- but in one cases the subject of the sentence follows the ring to the subject. Similarly, the consists not predicate (MVP post) while in the others the subject preonly of a noun and an article but is instead further en- cedes the predicate (MVP pre). The latter configuration, riched with another Prepositional Phrase . The in Italian, is more common that the former: we aim to gives more information about the relation be- investigate whether this syntactic variation can alter the tween the subject noun and the nominal part of the pred- performance of an LLM. icate. We define, for each pair of sentences, a question that
We exploit the diferent role of the two Prepositional asks the model to predict which element of the sentence Phrases to design a Q&A task on NC canonical and NC is involved in a certain action, either as the subject entity inverse sentences and hence assess whether a more com- or the object. In particular, for sentences that contain plex syntactic structure can influence LLMs capabilities. intransitive verbs, the model is always asked to predict Given an NC sentence, the model is asked to correctly the subject of the sentence, while in transitive cases (like interpret the meaning of the sentence by examining its the one in Table 1) the model is either asked to predict the predicate: in particular, the model is asked to predict subject or the object of the sentence. For this subdataset, the additional information related to the nominal predi- while the original data included both declarative and cate – which is included in the – by answering interrogative sentences, we retained only the declarative a “wh-” question (in Italian, "Di cosa", see the example ones: we test the model with a total of 192 sentence pairs. in Table 1). While both Prepositional Phrase answer to a To answer those questions, the relevant words – both wh-question, only the is related to the predicate for humans and LLMs – are the nouns that play the role of of the sentence and hence the model should be able to subject, or object if present, in sentences. In the next Secpredict the and ignore the . tion, we describe how it is possible to quantify whether
To solve the proposed task and to properly understand a model is able to identify the role of those words during NC sentences, humans and LLMs are required to im- the generation of the answer.
OC SVO NC inverse NC canonical MVP post MVP pre 2.2. Localizing Syntactic Knowledge via consider the tokens to be attributed for the generated Attributions answer produced by the model: for each correct answer generated by the model, we count the number of times Knowing which sentence structures are easier or more the tokens with the larger attribution value are the reledificult for a model to analyze is not enough. Consider- vant ones. This measures the accuracy of the attention ing the black-box nature of these models, it is essential head ℎ in recognizing the relevant words to generate the to understand which layers are responsible for encoding answer. syntax, thus making the models more interpretable. The more often the attention head focuses on the rel
We hypothesize that there is an area of the model evant words, the more syntactic knowledge the head responsible for correctly analyzing the sentence from the encodes. For each downstream task presented in Section syntactic point of view in order to get the answer to the 2.1, we collect the accuracy of all heads at all levels. Then, Q&A task. In fact, as discussed in the previous Section, we identify a head as "responsible" for generating the tarto answer correctly, the model needs to implicitly parse get word in a task if its score is higher than the average the roles of the words in the sentence and identify the score for that task. Specifically, we assume a Gaussian relevant words for the response (subjects and objects in distribution of scores for each task and identify a head the questions on OC, SVO and MVP sentences and the as responsible if the probability of observing a value at correct prepositional phrases in NC sentences). Hence, a least as extreme as the one observed is below a threshold knowledge of syntax is required to identify the relevant < 0.05. We also consider responsible all heads that words and, consequentially, generate the correct answer. obtain an excellent accuracy score (greater than 0.9) in
In generating the answer, we expect the model to “fo- focusing on the relevant words. With this procedure, for
cus” on those relevant words. We can identify to which each layer and task, we can localize the responsible heads
token the model focuses during generation, measuring and determine where the model encodes syntax the most.
token-to-token attributions [
library [21]
served in the previous subdataset emerge. In particular, the NC inverse sentences are harder than the corWe initially revise model’s accuracy on question compre- responding NC canonical: the average model accuracy hension task and assess models capabilities when difer- is 81.88(± 11.78) on NC canonical sentences, while the ent syntactic structures are involved (Section 3.1). Then, accuracy on NC inverse sentences is much lower, with we aim to spot the layers responsible for the correct syn- an average value of 64.06(± 28.26). Also in this case, tactic understanding of the sentences (Section 3.2). the results demonstrate that models are afected by different syntactic patterns. The model that better capture 3.1. Models accuracy on the right information to extract is modello-italia-9b on question-answering task both NC inverse and NC-canonical sentences. Although the performance of Llama-2-7b is rather low on inverse Results on each of the subdatasets show that the syntactic NC sentences (the model tends to generate very often structure of a sentence influences the models’ understand- the ), the remaining LLaMA-base models achieve ing of that sentence (see Table 2): across all tasks, LLMs better performance on both tasks. tend to obtain larger accuracy on sentences characterized Finally, results on the MVP task further confirm the by a unmarked syntactic structure. models’ behavior observed on the previous two tasks:
On the first task, on OC and SVO sentences, the mod- the inversion of the subject and verb positions causes els tend to struggle, especially in the OC sentences. On the models to perform worst on MVP post sentences OC sentences, some models, in fact, do not perform far (87.5(± 19.38) average accuracy) with respect to MVP from the random baseline of 50% accuracy ("yes" and pre (68.23(± 10.37) average accuracy). The average drop "no" answers are balanced). When comparing OC and in performance is larger than in previous subtasks: these SVO sentences, on average, the model accuracy drops results confirm that the inversion of the subject, even by 11.88(± 3.84) points when the sentence presents the in basic sentences, can degrade models’ understanding. object in the left-peripheral position. This result aligns Modello-italia-9b – probably due to the limited length with the dificulty that humans encounter in understand- of the input sentences – tends to replicate the input sening those sentences. The model that achieves the highest tences. The other models solve the tasks with excellent accuracy in this task in OC sentences is LLaMAntino- accuracy in the MVP pre sentences. 3-ANITA-8B, with an accuracy of 76.56. It is important to note that the model performance increase of 3.2. Localizing Layers responsible for 11.72 points with respect to the corresponding MetaLLama-3-8b (that achieves an accuracy of 64.84): these Syntax results stress the efectiveness of the finetuning for the Italian language. Across the LLaMa-based models the LLaMAntino-3-ANITA-8B is still the best performing model, followed by Meta-LLama-3-8b and with a larger gap by LLama-2-7b. The Qwen2-7B model is the best answering to the task on unmarked sentences.
On the NC sentences, similar patterns to the one ob
tures on model performance, we can identify the attention heads and levels of the models that mostly encodes syntax. In Figure 1 the number of responsible head at each layer of the models is reported for the Q&A task on NC sentences, (the remaining tasks are in Appendix A.3).
The general trend is that the most active in identifying relevant words during response generation layers are comprised between layer 19 and 25. Moreover, for all models, the layers we identify as responsible often handle multiple syntactic structures. The most noticeable result is that for the same task, the same activation trend emerges across all sentences.
A large number of responsible attention heads appear
around layer 19 to 27 in LLaMAntino-3-ANITA-8B and
Meta-Llama-3-8B. Layer 21, in particular, is the layer with
the most responsible heads both in NC and MVP tasks.
This layer is predominant also in the OC task,
concomitant with layers 19 and 22 (Figure 3a). For Llama-2, we
observe the same pattern as the most active layers are
between 18 and 25. On the Qwen2-7B model and
modelloitalia-9b active layers are higher in the architecture: from
layer 18 to 24 for Qwen2-7B (with layer 23 being the more
active in NC and MVP tasks) and from layer 21 to 31 on
NC and MVP senteces for modello-italia-9b. This finding
suggests a diferent interpretation of LLMs layers from
that previously observed in BERT [
While we could expect some correlation between the accuracy of the task and the capability of the model to identify the correct word in the sentence, the responsible heads appear to be shared across diferent syntactic structures. Those results suggest that some layers, more than others, encode syntactic information about the role of a word in a sentence. Moreover, diferent models and architectures seem to share a rather similar organization.
equivalent sentences are processed by LLMs in Italian when their syntax difers. We tested LLMs trained on the Italian - or with Italian data in the pre-trainig material - and measured how their capabilities in a battery of Q&A tasks that rely on parsing the correct role of words in a sentence to be solved. Our findings confirm that cleft sentences and construction with an inversion of subject and verb are dificult to understand also for LLMs - similarly to what observed for humans. Furthermore, we have identified systematically using token-to-token attribution that syntactic information tends to be encoded in the middle and top layers of LLMs. cessing (EMNLP), Association for Computational tational Linguistics, Online and Punta Cana, DoLinguistics, Online, 2020, pp. 7057–7075. URL: https: minican Republic, 2021, pp. 2888–2913. URL: https: //aclanthology.org/2020.emnlp-main.574. doi:10. //aclanthology.org/2021.emnlp-main.230. doi:10. 18653/v1/2020.emnlp-main.574. 18653/v1/2021.emnlp-main.230. [11] G. Kobayashi, T. Kuribayashi, S. Yokoi, K. Inui, [17] M. Abdou, V. Ravishankar, A. Kulmizev, A. Søgaard, Incorporating Residual and Normalization Lay- Word order does matter and shufled language moders into Analysis of Masked Language Mod- els know it, in: S. Muresan, P. Nakov, A. Villavicenels, in: M.-F. Moens, X. Huang, L. Specia, cio (Eds.), Proceedings of the 60th Annual Meeting S. W.-t. Yih (Eds.), Proceedings of the 2021 Con- of the Association for Computational Linguistics ference on Empirical Methods in Natural Lan- (Volume 1: Long Papers), Association for Computaguage Processing, Association for Computational tional Linguistics, Dublin, Ireland, 2022, pp. 6907– Linguistics, Online and Punta Cana, Domini- 6919. URL: https://aclanthology.org/2022.acl-long. can Republic, 2021, pp. 4547–4568. URL: https: 476. doi:10.18653/v1/2022.acl-long.476. //aclanthology.org/2021.emnlp-main.373. doi:10. [18] D. Brunato, C. Chesi, F. Dell’Orletta, S. Montemagni, 18653/v1/2021.emnlp-main.373. G. Venturi, R. Zamparelli, et al., Accompl-it@ [12] Y. Belinkov, J. Glass, Analysis methods in neural lan- evalita2020: Overview of the acceptability & comguage processing: A survey, Transactions of the As- plexity evaluation task for italian, in: CEUR WORKsociation for Computational Linguistics 7 (2019) 49– SHOP PROCEEDINGS, CEUR Workshop Proceed72. URL: https://aclanthology.org/Q19-1004. doi:10. ings (CEUR-WS. org), 2020.
1162/tacl_a_00254. [19] EVALITA 2020 — evalita.it, https://www.evalita.it/ [13] J. Hewitt, C. D. Manning, A structural probe for find- campaigns/evalita-2020/, 2020. ing syntax in word representations, in: J. Burstein, [20] M. Greco, P. Lorusso, C. Chesi, A. Moro, AsymmeC. Doran, T. Solorio (Eds.), Proceedings of the 2019 tries in nominal copular sentences: PsycholinguisConference of the North American Chapter of the tic evidence in favor of the raising analysis, Lingua Association for Computational Linguistics: Human 245 (2020) 102926.
Language Technologies, Volume 1 (Long and Short [21] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. DePapers), Association for Computational Linguistics, langue, A. Moi, P. Cistac, T. Rault, R. Louf, M. FunMinneapolis, Minnesota, 2019, pp. 4129–4138. URL: towicz, J. Brew, HuggingFace’s Transformers: Statehttps://aclanthology.org/N19-1419. doi:10.18653/ of-the-art Natural Language Processing, ArXiv v1/N19-1419. abs/1910.0 (2019). [14] G. Jawahar, B. Sagot, D. Seddah, What does BERT [22] Qwen2 technical report (2024). learn about the structure of language?, in: A. Ko- [23] M. Polignano, P. Basile, G. Semeraro, Advanced rhonen, D. Traum, L. Màrquez (Eds.), Proceedings natural-based interaction for the italian language: of the 57th Annual Meeting of the Association for Llamantino-3-anita, 2024. arXiv:2405.07101. Computational Linguistics, Association for Com- [24] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almaputational Linguistics, Florence, Italy, 2019, pp. hairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhar3651–3657. URL: https://aclanthology.org/P19-1356. gava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, doi:10.18653/v1/P19-1356. M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, [15] E. S. Ruzzetti, F. Ranaldi, F. Logozzo, M. Mastromat- W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, tei, L. Ranaldi, F. M. Zanzotto, Exploring linguistic A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Karproperties of monolingual BERTs with typological das, V. Kerkez, M. Khabsa, I. Kloumann, A. Koclassification among languages, in: H. Bouamor, renev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, J. Pino, K. Bali (Eds.), Findings of the Association D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, for Computational Linguistics: EMNLP 2023, Asso- P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenciation for Computational Linguistics, Singapore, stein, R. Rungta, K. Saladi, A. Schelten, R. Silva, 2023, pp. 14447–14461. URL: https://aclanthology. E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, org/2023.findings-emnlp.963. doi: 10.18653/v1/ R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, 2023.findings-emnlp.963. I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, [16] K. Sinha, R. Jia, D. Hupkes, J. Pineau, A. Williams, A. Rodriguez, R. Stojnic, S. Edunov, T. Scialom, D. Kiela, Masked language modeling and the dis- Llama 2: Open foundation and fine-tuned chat tributional hypothesis: Order word matters pre- models, 2023. URL: https://arxiv.org/abs/2307.09288. training for little, in: M.-F. Moens, X. Huang, arXiv:2307.09288.
L. Specia, S. W.-t. Yih (Eds.), Proceedings of the [25] iGenius | Large Language Model — igenius.ai, https: 2021 Conference on Empirical Methods in Natu- //www.igenius.ai/it/language-models, 2024. ral Language Processing, Association for Compu- [26] AI@Meta, Llama 3 model card (2024). URL:
token-to-token attribution to spot what is the most
relevant word during the generation of the answer in
LLMs on our task. The norm based approach is proposed
in Kobayashi et al. [
depicted. As described in Section 3.2, some layers tend to demonstrate a high number of attention heads responsible for the generation. In particular, layers around layer A.2. Prompts to Instruction-Tuned LLMs 20 seem to focus more on relevant words for the correct for the Italian Laguage generation of the answer than the other. Since the correct generation implies the capability of understanding Each model has been prompted with four diferent the role of diferent words by a model, we claim that prompts for each Q&A task (as described in Section 2.1). those level encodes some kind of syntactic information. Here is a complete list of the prompts template used in It is worth noticing that similar layers are responsible for our experiments: in the template the {Item} is the sen- the diferent sub tasks, in particular for the LLaMa-base tence to be analyzed and {Question} is replaced with models and for Qwen-2-7b model. the corresponding comprehension question.
OC and SVO senteces:
(a) OC and SVO sentences
(b) MVP sentences