Masked language models (MLMs) are deep neural language models which create contextualised word representations, in the sense that the representation for each word depends on the entire context in which it is used. That is to say, word representations are a function of the entire input sentence. Such models are designed to have high predictive capabilities and usually pre-trained on large textual corpora. This makes them closely tied to the domains on which they were trained and dependent on the infrastructure upon which they are based. The presence of various biases in MLMs has been extensively studied, typically with the aim of proposing effective mitigation strategies [1,2,3,4]. However, there are instances where the bias in certain MLMs is not necessarily negative. This is particularly true when the bias manifested in the language reflects its socio-temporal context. This bias could be advantageous for tasks that demand such socio-temporal staging [5,6].

In this paper, we explore the idea of analysing the bias by focusing on the major syntactic, semantic, and grammatical differences between two varieties of the English language: Early Modern (EME) and Modern (ME). More precisely, we propose a method and a measure of adequacy to test the adherence of MLMs to the natural language variety of interest. In particular, we assess the level of diachronic bias of three MLMs: Bert-Base-Uncased [7] (referred as BERT Base here) 1 ; MacBERTh [8] 2 ; and Bert British Library Books English [9] (BL Books) 3 . In our preliminary experiments, we perform tests with 60 masked questions in which the models have the task to predict the masked word in the sentence.

We then rate the proposed responses according to a 5-point bipolar scale between the two language variants and derive a weighted score from the response probabilities and their respective scores on the scale.

These results, although preliminary, might suggest a method applicable in the digital humanities when MLMs are employed for the analysis of historical corpora.

If it is true that language shapes culture while it is shaped by it [10], language models in general -and MLMs in particular -constitute a still partially covered mirror of this dual relationship. Not only can a MLM be tested based on its level of representativeness of the language to determine its reliability, but also it can tell us about linguistic, social, and historical phenomena that concern the culture tied to that specific language. In other words, a MLM could be a valuable tool towards the expansion of the broader social knowledge of a given culture, rightfully becoming part of the basic tools of Cultural Analytics discussed by Manovich [2020]. According to Bruner's [1984] pragmatic-cultural perspective, learning a language also means learning the cultural patterns associated with it. Similarly, analysing the language in its various realisations would mean having the opportunity to visualise the underlying cultural patterns.

Moreover, MLMs can be highly beneficial also for philological [13], pragmatic [14], critical [6], and literary work [15]. However, the effectiveness of these models depends on their ability to adapt to language specificity in its historical dimension. This is typically achieved by training models on historical text corpora. However, the difficulty of accessing large historical documentary collections means that the models available are still few and requires verifying whether they adapt effectively to the historical linguistic context.

BERT is a foundational masked language model (MLM) which to date is the most widely adopted [16]. A number of studies have explored different forms of bias in BERT [1,2,3]. Three BERT-based MLMs are of particular interest for our study: (i) Bert-Base-Uncased [7], created from a corpus of texts from Wikipedia and BookCorpus and a model of contemporary language, which we use as a control condition in our experiment; (ii) MacBERTh [8], pre-trained on texts from 1500 to 1950; and (iii) Bert British Library Books English, pre-trained on contemporary texts and fine-tuned on historical texts from the 19th century to the present.

To evaluate the adequacy of MLMs on a test set, we define a temporal valence task consisting of a collection of test sentences, each with a masked token (i.e., word). This is a typical fill-in-the-blank task, where the models are required to predict the masked token. Formally, we consider the following three sets: (i) we denote with 𝒮 the set of all test sentences, (ii) with 𝒱 we denote a set of vocabulary words, and (iii) with 𝒯 = {−1, −0.5, 0, 0.5, 1} ⊂ R, we denote a 5-point bipolar temporal valence scale, where −1 represents the farthest historical period and 1 the closest to today.

With the above notation, for each of the masked sentences (denoted as 𝑠 ∈ 𝒮), we define a function 𝜌 : 𝒮 → 𝒯 representing the sentence temporal valence score. This function indicates the period from which the masked sentence is typical.

Then, we calculate a token-in-sentence temporal valence score 𝜎 : 𝒱 × 𝒮 → 𝒯 , indicating the score of a token substituting the sentence mask.

The mentioned temporal valence scores are assigned arbitrarily according to the research hypotheses. Taking this study as an example, the criterion used to determine each score was the degree of alignment of certain sentences or tokens with a specific historical period on a philological-linguistic basis. Scholars wishing to delve into language study using this methodological approach can selectively choose the score to assign to their test set based on their specific research needs. The versatility of the proposed methodology is evident in its adaptability to a diverse array of fields of interest. This flexibility enables researchers to seamlessly integrate personalized metrics, ensuring a tailored approach to analysis without undermining the inherent consistency of the results.

As an example of temporal valence score, given EME (Early Modern English) as the farthest period (i.e., −1 ∈ 𝒯 ) and ME (Modern English) as closest (i.e., 1 ∈ 𝒯 ), if we consider the sentence 𝑠 1 = "Why wilt [MASK] be offended by that?" we have 𝜌(𝑠 1 ) = −1 as 𝑠 1 is a representative sentence for EME, and 𝜎("𝑡ℎ𝑜𝑢", 𝑠 1 ) = −1, because in this context "thou" is indicative for EME. On the other hand, 𝜎("𝑛𝑜𝑡", 𝑠 1 ) = 0, because "not" is neutral regarding the two language varieties. Given a model 𝑚, for the masked token in each sentence (𝑠 ∈ 𝑆), we have the set of {𝑤 1 , 𝑤 2 , . . . , 𝑤 𝑛 } ⊂ 𝒱 of 𝑛 words predicted by 𝑚 for 𝑠, that are associated with the vector of corresponding probabilities from this model, shown in Equation 1.

For this set, using the temporal valence score 𝜎, we define a token-in-sentence temporal valence score vector x 𝑚 for 𝑚 given the sentence 𝑠 as in Equation 2.

x 𝑚 = (𝜎(𝑤 1 , 𝑠), 𝜎(𝑤 2 , 𝑠), . . . , 𝜎(𝑤 𝑛 , 𝑠)) 𝑇

This allows us to define the bias of a model regarding the sentence as the dot product of the modelderived probabilities and the token valence scores, providing us with a weighted score as in Equation 3, and effectively getting a single value measurement from the two vectors above.

We can also proceed to define the domain adequacy of a model with respect to a sentence 𝑠 (see Equation 4), based on the difference between the sentence temporal valence score 𝜌(𝑠) and the model bias 𝛽(𝑚, 𝑠). To do this, we consider the difference between the model bias and the sentence temporal valence (disregarding which one is larger), and project it on the unit interval, making sure that more similar values lead to higher adequacy scores.

Examples of three sentences classified in different periods are provided in Tables 1, 2 and 3, which show the corresponding values for 𝜌, 𝑝, 𝜎, 𝛽(𝑚, 𝑠) and 𝛿(𝑚, 𝑠).

We test our metrics with three BERT-based linguistic models we consider relevant for the varieties of the English language of interest: (i) Bert-Base-Uncased, (ii) MacBERTh, and (iii) BL Books. In accordance with the objectives of this study, the choice of models reflects a specific interest in language; therefore, they can be replaced to best fit any other specific interest in diachronic language analysis. For the test we used 60 word-masked sentences, specifically created for this study. To create the test set, we relied on different types of written language: contemporary standard, journalistic language, social media non-standard, and Early Modern language.

The elements to be masked were selected based on their belonging to specific word classes known to have suffered more exposure to the diachronic variation of the English language: pronouns, verbs, adverbs, adjectives, and nouns. Of the 60 sentences4 , 20 are selected to be suggestive for the EME variety of English, further 20 -as suggestive for ME, and final 20 are generic. Once the test set was complete, a temporal valence score was assigned to each sentence (see 𝜌 in Section 3) based on their level of chronological markedness.

The test set was administered to the three MLMs, and the suggested words with their probability were collected. The resultant vocabulary was marked independently from the models that provided it by setting the token-in-sentence temporal valence score (i.e. 𝜎) to each word, based on an estimation of proximity of the token's meaning to a certain linguistic variety in the context in which it appeared. Notably, during this phase, our decision was to work on a sentence level (contextually) rather than on a set level (globally). The method proved highly effective in avoiding the risk of semantic flattening, given that almost every word has shown some level of contextual semantic specificity if taken contextually rather than globally. An example is the pronoun you in "fare you well, sir", which is globally neutral and yet acquires a strong diachronic value if evaluated in its context, in which it appears to be utmost archaic.

Once 𝛽 and 𝛿 were calculated, we proceeded with the analysis of the data and the collection of results. The distribution of the bias score 𝛽 and the domain adequacy score 𝛿 for the sentences in the three groups (i.e., EME, Neutral, and ME) is shown in Figures 1 and 2, respectively. Figure 3 shows that for all three test sets, MacBERTh is most aligned with EME, whereas BERT Base is always most aligned as ME. BL Books shows a tendency towards a more neutral language than the other two models in marked sentences, whilst surprisingly it aligns to ME in neutral sentences. Figure 2 shows that MacBERTh has best domain adequacy for EME, and BERT Base has best domain adequacy for ME. In the case of the neutral test set, domain adequacy is no less informative. Although the sentences do not inherently carry their expectations regarding language, models appear consistently well-suited to a neutral context, and none of them pushes for strong specificity of their corresponding trained domain. In effect this leaves the sentences close to their original neutrality This preliminary study provides an illustration of the nature and functioning of the MLMs predictive behaviour. The presence or absence of markedness in the sentences enables all three MLMs to select the type of element which best fits the co-text. So, while for diachronically marked sentences, models without training in that domain attempted to suggest probable solutions, sometimes resulting in a form of linguistically inconsistent mimicry, in unmarked sentences, the models perform exceptionally well, and linguistic inaccuracies are rare.

Both notions of bias (𝛽) and domain adequacy (𝛿) provide important insights of the nature of the models. The first, 𝛽, indicates a tendency in terms of temporal valency. In other words, the interpretation of its value should be considered within the context of the specific dichotomy of language varieties. On the other hand, 𝛿 reflects the adequacy for an individual language variety. It successfully captures model tendencies when completing historically predetermined sentences. However, due to its inclusion of 𝜌, 𝛿 is less informative when there is no bias originating from the sentence, i.e. when completing temporally-neutral sentences.

Notably, our measures demonstrate that MacBERTh is better at representing the EME historical context than BL Books. This could possibly be explained by the nature of the models in question. First, MacBERTh is a model created from scratch and trained on texts spanning a time range that takes into account the evolution of the English language from EME to ME. BL Books, on the other hand, was only fine-tuned on texts from the modern period, so it has no direct exposure to EME. It does perform better on ME than MacBERTh and worse than BERT Base. Thus, MacBERTh demonstrates a strong linguistic consistency, given the wide range of language varieties it is trained on, but in tasks related to ME yields worse results than other more specialised models. Simply put, having a specific, narrower domain poses fewer problems when working within it but reveals clear gaps when moving outside that domain.

The notion that LMs can serve as a window into the history of a population is not new, but there is a growing interest in exploring the relationships between these models and the socio-linguistic and socio-cultural contexts [17,18,19,20]. It is equally imperative to establish a procedural framework to address the lack of evaluative methods for these models, as previously hinted at in this text. This is particularly useful when no direct links could be drawn between the corpus used to train the model and the social context of the test set.

Within this evaluation, we created a dedicated test set for each model under scrutiny, drawing upon approaches used for evaluation of bias in MLMs. In creating our test sets, we built our sentences both on logical-semantic and logical-syntactic tasks. Future work could try to create a test set for model interrogation that is culture-oriented, delving into socio-culturally significant elements such as customs, historical events, and attitudes towards social groups -elements recognised as belonging to social knowledge. Alternatively tests could be derived from word in context datasets, such as TempoWiC and HistWiC [21,22,23].

Alternatively, the temporal valence of word tokens could be derived not simply from the sentence where they emerge, but from the wider historical context, e.g. from a large corpus, representative for the period [24,25]. This would allow automating not only the calculation of the token temporal valence 𝜎, but also the identification of sentences that are representative for each historical period. As a consequence, the dependence on manual expert evaluation would be strongly reduced, which would result in both higher reproducibility and wider generalisability of the approach.

This study aims not only to propose a methodology for assessing language models but also to put forth hypotheses for expanding the available tools to humanities scholars interested in studying complex socio-cultural phenomena with an approach which begins by interpreting textual clues and inferring their connections to reality. As such it is also applicable to contexts beyond diachronic, but also across dialects or professional jargons.