This paper falls within ongoing research aimed at enhancing the human interpretability of neural language models by incorporating physiological data. Specifically, we leverage eye-tracking data collected during reading to explore how such information can guide model behavior. We train a multilingual encoder model to predict eye-tracking features from the Multilingual Eye-tracking Corpus (MECO) and analyze the resulting shifts in model attention patterns, focusing on how attention redistributes across linguistically informed categories such as part of speech, word position, word length, and distance from the syntactic head after fine-tuning. Moreover, we test how this attention shift impacts the representation of the interested words in the embedding space. The study covers both Italian and English, enabling a cross-linguistic perspective on attention and representation shifts in multilingual encoders grounded in human reading behavior.
attention patterns, as well as their performance on downstream tasks and representation space. Their findings Neural language models (NLMs) now match or even sur- show that this intermediate process increases the correlapass human benchmarks on many NLP tasks, yet the tion between model attention and human attention and it logic behind their predictions remains largely hidden be- leads to a compression of the embedding space, without hind billions of parameters. To make these systems more generally degrading performance on downstream tasks. transparent and data-eficient, researchers are increas- Building on this foundational framework, this paper ingly borrowing ideas from cognitive science, grounding aims to further highlight the efects of incorporating inboth training and evaluation in how people actually learn formation about human reading behavior in a NLM and process language (e.g. [1, 2, 3]). Among the most from a linguistically informed perspective. Specifiinformative cognitive signals of human language pro- cally, we examine how fine-tuning on eye-tracking sigcessing is eye-tracking (ET). Decades of psycholinguistic nals leads to shifts in model attention, and how these work show that fixation times, regressions, and skips shifts afect the structure of word representations. To mirror both early lexical access and later integrative pro- explore this, we extract a set of linguistic features, capturcesses underlying text comprehension [4, 5]. Leveraging ing progressively more complex language phenomena, these signals has already boosted model accuracy on a from the input text and analyze how attention is redisvariety of downstream tasks ranging from core linguistic tributed across word classes defined by these features. In tasks [6] to more applied tasks like sentiment analysis [7], parallel, we assess how these attention shifts influence language proficiency assessment [ 8], machine reading the embedding space, both at a global level and within the comprehension [9], while also giving us a new lens on local representational geometry of specific word classes. model interpretability. Studies by Sood et al. [10] and The code for our experiments is publicly available on Eberle et al. [11] found that transformer attention does GitHub. not always line up with human gaze, whereas Bensemann et al. [12] and Wang et al. [13] revealed stronger links in specific layers, hinting at a layered correspondence 2. Related work between reading behavior and neural representations.
Extending this direction, Dini et al. [14] investigate how injecting reading-related information into NLMs through diferent fine-tuning strategies on ET data afects their
within NLMs interpretability. The first investigates ET data as a diagnostic signal to evaluate the alignment between model behavior and human cognitive processing, particularly through the lens of attention mechanisms. The second focuses on analysing model’s attention mechanisms (Section 2.2) and representational space (Section
entity’s determiners or subjects attending to their verbs.
However, fine-tuning on syntactic or semantic tasks had minimal efect on altering self-attention patterns. Vig and Belinkov [26] conducted a comprehensive analysis of attention head interpretability in GPT-2 using both visualization and quantitative measures. Their results indicate a layer-specific linguistic sensitivity, with diferent types of linguistic information—such as PoS and syntactic dependencies—being more salient in particular layers.
They also found stronger alignment with syntactic dependencies in the model’s middle layers. Htut et al. [27] directly evaluated the extent to which attention aligns with gold-standard dependency parses. By computing the correspondence between attention distributions and syntactic head-dependent pairs, they showed that BERT’s attention does not systematically reflect syntactic dependency structures, particularly in deeper layers.
Taken together, these studies suggest that while attention mechanisms can exhibit linguistically meaningful behavior in isolated cases—especially in specific layers or individual heads—they do not consistently encode syntactic or morpho-syntactic structure.
In recent years, eye-tracking has emerged as a prominent physiological signal in NLP research due to its afordability and ease of collection compared to methods like fMRI or MEG. Public resources such as the GECO corpus [15], the MECO corpus [16], and the WE-RDD dataset [17] now let researchers probe gaze behaviour at scale across languages and reading paradigms.
Work with these corpora has split in two directions.
The former injects gaze-derived features, into neural architectures, typically lifting accuracy on downstream tasks. The latter, which motivates our study, treats ET as a diagnostic for a model’s internal workings.
The first systematic comparison came from Sood et al. [18], who matched attention maps from CNNs, LSTMs and Transformers against human fixations. Their findings reveal that while transformers performed the best, they showed the weakest alignment with gaze. Eberle et al. [11] confirmed that even after task-specific finetuning, large Transformers stayed distant from human reading patterns. Conversely, Bensemann et al. [12] reported that raw dwell times correlate strongly with the earliest BERT layers, a relation that persists as model size grows. Morger et al. [19] extended the inquiry cross- 2.3. Geometry of the embedding space lingually and found robust correlations, especially for Transformer models learn a high-dimensional embedding monolingual encoders, between human word-importance space in which every token is represented by a dense vecrankings and model saliency. Most recently, Wang et al. tor that encodes both meaning and syntax. A consistent [20] showed that deeper layers of NLMs once again echo ifnding is that these vectors occupy only a narrow cone ifxation metrics, hinting at a layered, non-monotonic link of the space, an anisotropic layout sometimes called the between model depth and cognitive fidelity. representation degradation efect [ 28, 29, 30]. In NLP, such behaviour is often viewed as harmful because it 2.2. Model Attention Dynamics can hide fine-grained linguistic cues [ 31, 32, 33]. Yet theory and broader machine-learning evidence show that The role of attention mechanisms in NLMs has been a anisotropy can arise naturally under stochastic gradisubject of extensive research and debate. While atten- ent descent and may even aid generalization, especially tion weights are often interpreted as providing insight when models project data onto low-dimensional maniinto model reasoning, a growing body of research has folds [34, 35, 36, 37]. In this respect, studying the impact questioned their reliability as faithful explanations of of various fine-tuning objectives and downstream tasks model decisions. Some studies suggest that attention provides important insights into how they shape the gecan highlight important input elements, yet others argue ometry of the embedding space [34, 35, 36]. While still that attention distributions can be manipulated with- relatively limited, a growing body of work has begun out significantly afecting predictions, casting doubt on to examine the relationship between embedding space their explanatory power [21, 22]. In response to these properties and linguistic phenomena. For example, Herconcerns, alternative attribution methods have been pro- nandez and Andreas [38] show that linguistic features posed—such as attention rollout [23] and gradient-based tend to be encoded in lower-dimensional subspaces in techniques [24]—which aim to better capture the path- the early layers of both ELMo and BERT and that relaways through which information influences predictions. tional features (like dependency relations between pairs As part of this debate, a parallel line of work has ex- of words) are encoded less compactly than categorical plored whether attention aligns with known linguistic features like part of speech. More recently, Cheng et al. structures, such as syntactic dependencies or PoS cate- [39] analyzed representation compression in pre-trained gories, ofering a complementary perspective on its in- language models from both geometric and informationterpretability. The foundational study by Clark et al. [25] theoretic perspectives. Their findings reveal a strong showed that certain attention heads in BERT consistently correlation between these two views and show that the focus on syntactic phenomena, such as attending to an intrinsic geometric dimension of linguistic data is predictive of its coding length under the language model.
To the best of our knowledge, no systematic study has examined how eye-tracking fine-tuning afects attention patterns and the resulting embedding representations across diferent linguistic phenomena. Moreover, crosslinguistic analyses of these changes following cognitively motivated fine-tuning remain scarce.
Multilingual Eye-tracking Corpus (MECO) to finetune the model on human gaze modeling and treebanks from the Universal Dependencies (UD) project to extract linguistically motivated features and compute model attention shifts and representation structure induced by ifne-tuning on ET data.
contains ≈ 13, 000 sentences drawn from a variety of textual genres. For English, we used the training set of the English Web Treebank (EWT) [41], including ≈ 12, 000 sentences , also multi-genre. UD corpora were chosen due to their gold-standard syntactic and part-of-speech annotations, which provide a reliable foundation for our ifne-grained linguistic analyses. Additionally, the crosslinguistically consistent annotation schema ofered by UD enables meaningful comparisons across typologically distinct languages.
We propose a linguistically informed framework to investigate the impact of injecting human reading behaviour into a pre-trained NLM, focusing on its efects on attention and word representations. The approach 3.1. Eye-tracking data: The MECO Corpus consists of two main stages: first, we fine-tune the model on predicting several ET features; then, we compare the MECO [16] is a multilingual collection featuring read- pre-trained and fine-tuned models along three axes: i) ing behavior from both native (L1) and second-language Correlation between model attention and human attenspeakers across 13 languages. We focus on the L1 subsets tion; ii) Attention distribution over input tokens; iii) Senfor English and Italian, chosen for their typological di- tence representations in the embedding space. versity and data completeness, allowing for a controlled To enable a more fine-grained analysis of how ET fineyet cross-linguistic perspective on gaze modeling. tuning afects word representations, we condition our
Each participant in MECO read 12 encyclopedic-style evaluations on the following linguistic features extracted texts, covering general knowledge topics. To ensure con- from the UD treebanks: word length in characters, part sistency and limit computational costs, we selected the of speech category, position in the sentence, and dislargest subsets of users who had read the majority of sen- tance from the syntactic head. tences. For Italian, we included 9 participants who read For our experiments we used XLM-RoBERTa-base, a 12 all sentences. For English, since no participant completed layer multilingual encoder-based model. In what follows, the full set, we selected 25 participants who all read the we outline the methodological choices and implementasame set of sentences, missing only two in common. tion details of our experimental setting.
We used five ET features intended to represent early, late and contextual signals of human reading processes: 4.1. ET injection into the Model First Fixation Duration: the duration of the first fixation landing on the word; Gaze Duration: the summed duration of fixations on the word in the first pass, i.e., before the gaze leaves it for the first time; Total Reading Time: the cumulative amount of time spent reading a word, capturing both fixations and potential interruptions (e.g., regressions or pauses); First-run Number of Fixations: the number of fixations on a word during the ifrst pass; Total Number of Fixations: the number of discrete fixations on areas of interest overall.
leverage the set of eye-tracking features from MECO described in Section 3.1. Unlike most prior work—which typically aggregates eye-tracking data across participants, with few exceptions [42]—we treat each reader individually, conducting experiments separately for each subject.
This design choice is motivated by the intrinsic variability observed in reading behavior, even among skilled readers [43, 44, 45], and enables a more accurate modeling of reader-specific dynamics. 3.2. Universal Dependencies Treebanks After a hyperparameter tuning phase using 5-fold cross-validation, we fine-tune the model to predict To analyze how model attention weights and embedding five word-level eye-tracking features , training a sepspace shift following fine-tuning on eye-tracking data, arate model for each individual reader. we relied on linguistically annotated corpora from UD Since the MECO dataset provides annotations at the treebanks [40]. Specifically, for Italian, we employed word level, while the model’s tokenizer splits some words the subsection corresponding to the training set of the into subword units, we follow standard practice [46] and assign eye-tracking features only to the first sub-token of each word, ignoring the rest during training1.
To examine whether the fine-tuned model develops a more human-like attention pattern, we compute the correlation between model attention and human attention before and after fine-tuning. For model attention, we consider the attention weights received by each word when computing the representation of the beginning-of-sentence token (<s>), which is the only token used during the eye-tracking prediction phase and serves as a global summary of the sentence. To account for subword tokenization, we follow the same approach used during fine-tuning and associate attention scores to the first sub-token of each word. As a proxy for human attention, we choose the Total Reading Time feature (see Section 3.1). For each reader, we thus compute the correlation between their eye-tracking data and the attention patterns of both the pre-trained and the fine-tuned model across all layers, allowing us to assess whether the latter aligns more closely with human reading behavior.
of each word in the UD treebanks. In line with the other analyses, we compare the embedding spaces of the pretrained and fine-tuned models to assess whether ET finetuning leads to more compact or more isotropic representations, as reflected by changes in these metrics.
All reported scores are first computed for each user individually and then averaged across all users.
Specifically, to compute the attention shifts, for each value of these features, we analyse the amount of attention the corresponding words receive before and after ifne-tuning. This allows us to characterize shifts in attention distribution across diferent linguistic phenomena and across all layers of the models. Firstly, we normalize the attention scores for each sentence (excluding BOS and EOS tokens) so that their sum is 1. Attention shifts are quantified as the percentage change in the average attention received by tokens with a given feature value, Figure 1: Correlation between model attention and human before fine-tuning. A positive shift indicates increased attention (p-value < 0.05). attention to these tokens after fine-tuning, while a negative shift reflects a decrease. This allows us to identify As a first evaluation step, we computed the correlation which linguistic categories gain or lose prominence after between human attention and model attention, both beincorporating eye-tracking supervision. fore and after fine-tuning on eye-tracking data. As we
To analyze the shifts in the embedding space, we are interested in the strength rather than the direction rely on two complementary metrics. (i) IsoScore [47] of- of the association, we considered the absolute values of fers a scale-invariant measure of isotropy: lower scores the correlation coeficients. For the fine-tuned models, indicate that the embedding variance is concentrated we computed the correlation between the model’s attenalong fewer directions, pointing to a more anisotropic tion weights and the Total Reading Time of the specific space. (ii) Linear Intrinsic Dimensionality (Linear-ID) [48] user on which each model was fine-tuned. For the preestimates the dimensionality of the smallest linear sub- trained model, which is not finetuned to any individual space that captures the embeddings, providing a proxy reader, we calculated the correlation between its attenfor their geometric complexity. tion weights and the Total Reading Time of each user independently, and subsequently averaged the resulting 1The fine-tuning is run for 50 epochs, using a learning rate of 5− 05, coeficients. Figure 1 reports the comparison of Spearman a weight decay of 0.01, and a warm-up ratio of 0.05. correlation coeficients, averaged across all users.
In line with results reported in [14, 49], fine-tuning on ET data consistently leads to stronger correlation coeficients between model and human attention, particularly in the deeper layers of the model. This efect is evident in both Italian and English. The overall patterns are remarkably similar across the two languages, although the correlation scores for Italian are slightly higher on average.
duced by fine-tuning on ET data. We grouped tokens into classes for the values of the linguistic features detailed in Section 4. To enhance readability and interpretability, for each linguistic feature we visualised only the most representative values. Rather than applying a strict frequency threshold, we heuristically excluded rare or degenerate cases (e.g., for token length, extremely long tokens such as URLs), retaining typical and frequent values that better reflect standard linguistic patterns. Each figure also includes an “AVG” column summarizing the average shift across all layers, ofering a high-level view of the attention reallocation patterns. and sentence interpretation. Additionally, a language- tive data from the used UD treebanks show that early specific efect is visible in Italian, where coordinating sentence positions largely correspond to syntactically conjunctions (CCONJ) gain notable attention across sev- central elements—particularly the root, which anchors eral layers. While similar shifts occur sporadically in the the clause and governs the structure of major compleEnglish model, they are less consistent and often ofset ments. The observed shift in attention may therefore by decreases in other layers. reflect the model’s increased sensitivity to syntactic or
As regards the attention shifts based on the word’s ganization cues at sentence onset, especially in specific position within the sentence (Figure 4), we noted that layers. This behavior is also well-documented in psyfor both languages tokens appearing earlier in the cholinguistic studies and indicative of incremental parssentence generally receive slightly more attention ing, where early elements guide syntactic and semantic after fine-tuning , whereas those occurring later receive expectations during sentence comprehension. less. An exception is observed for the first two tokens, Figure 5 shows the attention shifts for the headwhich deviate from this trend. Layer-specific behaviors dependent distance parameter. A positive value indicates also emerge: for instance, layers 2 and 9 tend to increase that the head follows the dependent, while a negative one attention toward later tokens, while most other layers that the head precedes it. The special value 0 is assigned show the opposite efect, emphasizing earlier positions. to the root of the sentence. On average, it emerged that Notably, layer 2 and layer 11 both show sharp increases tokens that are syntactically closer to their head in attention to the first token, suggesting a potential tend to receive more attention after fine-tuning, reweighting of sentence-initial information after expo- particularly when the head follows the dependent. sure to human reading patterns. Interestingly, quantita- This suggests that fine-tuning on ET data encourages
• FAIR - Future AI Research (PE00000013) projects under the NRRP MUR program funded by the
NextGenerationEU.;
• The project “XAI-CARE” funded by the
European Union - Next Generation EU - NRRP M6C2
“Investment 2.1 Enhancement and
strengthening of biomedical rese
F83C22002470001) • The project “Human in Neural Language Models” (IsC93_HiNLM), funded by CINECA3 under the
ISCRA initiative;
• Language Of Dreams: the relationship
between sleep mentation, neurophysiology,
and neurologi
Revisiting representation degeneration problem in 1162/coli_a_00402. doi:10.1162/coli_a_00402.
language modeling, in: Findings of the Association [41] N. Silveira, T. Dozat, M.-C. de Marnefe, S. Bowman,
for Computational Linguis
Natural Language Processing, Association for [43] A. J. Parker, T. J. Slattery, Spelling ability influences Computational Linguistics, Miami, Florida, USA, early letter encoding during reading: Evidence from 2024, pp. 5999–6011. URL: https://aclanthology.org/ return-sweep eye movements, Quarterly Journal of 2024.emnlp-main.344/. doi:10.18653/v1/2024. Experimental Psychology 74 (2021) 135–149. URL: emnlp-main.344. https://doi.org/10.1177/1747021820949150. doi:10. [35] W. Rudman, C. Eickhof, Stable anisotropic reg- 1177/1747021820949150, pMID: 32705948. ularization, in: The Twelfth International Con- [44] J. Ashby, K. Rayner, C. Clifton, Eye moveference on Learning Representations, 2024. URL: ments of highly skilled and average readers: Difhttps://openreview.net/forum?id=dbQH9AOVd5. ferential efects of frequency and predictability, The Quarterly Journal of Experimental Psychology Section A 58 (2005) 1065–1086. doi:10.1080/ 02724980443000476. [45] T. J. Slattery, M. Yates, Word skipping: Efects of word length, predictability, spelling and reading skill, Quarterly Journal of Experimental Psychology 71 (2018) 250–259. doi:10.1080/17470218.2017.
1310264. [46] N. Hollenstein, F. Pirovano, C. Zhang, L. Jäger,
L. Beinborn, Multilingual language models predict human reading behavior, in: K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, Y. Zhou (Eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online, 2021, pp. 106–123.
URL: https://aclanthology.org/2021.naacl-main.10/.
doi:10.18653/v1/2021.naacl-main.10.
[47] W. Rudman, N. Gillman, T. Rayne, C.
Eickhof, IsoScore: Measuring the uniformity of
embedding space utilization, in: S. Muresan,
P. Nakov, A. Villavicencio (Eds.), Findings of the
Association for Comput
18653/v1/2022.findings-acl.262. [48] J. H. Lee, T. Jiralerspong, L. Yu, Y. Bengio, E. Cheng,
Geometric signatures of compositionality across a language model’s lifetime (2025). URL: https://arxiv.
org/abs/2410.01444. arXiv:2410.01444. [49] L. Dini, L. Moroni, D. Brunato, F. Dell’Orletta, In the eyes of a language model: A comprehensive examination through eye-tracking data, Neurocomputing (2025). In press.
5.3 conducted on the remaining linguistic features: word length, Figures A.1 and A.2, and word index in sentence, Features A.3 and A.4. As in Section 5.3, a clear hierarchy emerges among the new feature classes. For word length, tokens 6–10 characters long retain the highest IsoScore and Linear-ID before collapsing, like all other bins, under ifne-tuning.
Figure A.2: Linear-ID before (left) and after (right) finetuning, shown for word length (up to 15 tokens).
Figure B.3: Isotropy before (top) and after (bottom) finetuning, grouped by syntactic head distance (up to 7 words of distance).
Figure B.4: Linear-ID before (top) and after (bottom) finetuning, grouped by syntactic head distance (up to 7 words of distance).
Declaration on Generative AI During the preparation of this work, the author(s) used ChatGPT (OpenAI) in order to: Improve writing style and Formatting assistance. After using these tool(s)/service(s), the author(s) reviewed