The aim of this short paper is to extend the application of embedding-based methodologies beyond the realm of lexical semantic change. It focuses on the use of unsupervised BERT-embeddings and uncertainty measures (Classification Entropy), and assesses whether (and how) they can be used to (semi-)automatically flag possible functional-semantic changes in the use of the construction [BE about] in the Corpus of Historical American English (COHA).
Given its long tradition in computational and statistical research, it comes as no surprise that
text-based humanities have embraced the use of distributional-semantic ‘vectors’ or
‘embeddings’ – i.e. (compressed) numeric vector representations of a word’s contextual distribution
that serve as a proxy of that word’s meaning [e.g. 3]. In particular, in fields such as Corpus
Linguistics – a subfield of Linguistics which grew around the computer-aided retrieval,
annotation, and later also categorization of textual data [
In recent years, we have also witnessed a growth in the number of studies that have utilized
either “count” or “predictive” vector models [
One consequence of this focus on lexical change is that, at present, the number of
computational distributional semantic studies that consider the functional-semantic properties of
more abstract, grammatical constructions seems disproportionate compared to the interest in
the phenomenon within the (Corpus) Linguistic community. Much like lexical semantics, the
function(s) and underlying meaning(s) of grammatical structures – which are often notoriously
polysemous and cover a broad range of nuanced, abstract meanings – are prone to change, and
many linguists believe that the continued discovery, description and analysis of such changes
plays an essential role in fleshing out our understanding of the mechanisms and motivations of
language change. A logical and necessary continuation in the pursuit to automated semantic
(shift) detection in large diachronic corpora would therefore involve further, more in-depth
explorations of the extent to which embeddings can be employed to capture the (changing)
functional-semantic properties of grammar. Given that that diferent components of language
have difering diachronic dynamics [
The present study, then, sets out to do precisely that: explore some possible avenues of
(semi)automatically detecting diachronic functional-semantic changes of grammatical constructions.
In that sense, the aim of this study is to extend the application of embeddings-based
methodologies beyond the realm of lexical semantics, and further add to the budding research on
whether and how token vectors [e.g. 14, 29] or contextualized embeddings [e.g. 4] can be used
to study the functional-semantic properties of grammar. More specifically, the study, which
focusses on embeddings created by means of BERT [
As a case study, I focus on the recent diachronic development of [BE about]. The data for this
study has been gathered from COHA, a corpus containing over 400 million words of American
English text written between 1810 and 2009. The corpus is balanced for genre (Fiction,
NonFiction, Magazine, Newspaper (after 1860)) and subgenre (e.g. prose, poetry, drama, etc.
(Fiction)). Such (sub-)genre balance is said to ensure that any changes observed in the corpus
will not simply be “artifacts of a changing genre balance”[
While [BE about] can be used with a wide range of more abstract, grammatical meanings
(which vary substantially in frequency as well as distinctiveness), the present study will focus
on its three most frequent usage types: the futurate (e.g. I am about to leave), approximative
(e.g. There were about ten cats in the room), and descriptive use (e.g. This song is about
love). These usage types align with the higher-order sense categories distinguished in the
Oxford English Dictionary (OED) [
Figure 1 shows a two-dimensional t-SNE future approx descriptive other mapping of the embeddings of 1,000 examples 40 of [be about], collected from COHA (decade 2000-2009). The examples were manually an- 20 notated at the level of granularity outlined 0 above. The sample contained 448 futurate, 20 279 approximative, and 225 descriptive uses 40 of [BE about]. The other 48 examples include irrelevant structures (e.g. due to mistakes in 60 40 20 0 20 40 60 COHA’s tagging of the possessive marker ’s as a finite form of the verb be), as well as three much more infrequent usages of [BE about]. Figure 1: t-SNE of [BE about] token embeddings. These infrequent types consist of spatial uses There are three largely distinct usage types: futurate, (e.g. She must be somewhere about), the fixed approximative, and descriptive. Less frequent senses expression that’s about it, as well as an ac- (e.g. spatial) are marked as ‘other’. tional use which can be paraphrased as ‘occupied/dealing with’ (chiefly found in more or less fixed expressions such as be about your business and know what one is about).
To assess whether BERT-embeddings can be used to identify and distinguish the three
usage types under scrutiny, we can conduct a simple ‘sense distinction task’. For this task,
the embeddings of the 952 ‘relevant’ examples and their accompanying usage type labels were
used as the training set. The procedure involved fitting a logistic regression classifier with L2
regularization (as implemented in Scikit-learn [
Based on the manual assessment of the pre- 1.00 dicted usage type labels, it appears that the 0.98 model performs quite well at distinguishing 00..9946 tFhoer tthherefinealmdaeicnaduesaogfeCtOypHeAs , oofnl[yBE5 oaubtouotf]. trrccoe000...989280 200 tokens had been mislabelled, indicating a 0.86 ccllaassssiifificcaattiioonn aaccccuurraaccyy ooff 0t.h9e75m. oNdoetlarbelmy,atihnes 0.84 1800 1825 1850 1875 1b9i0n0 1925 1950 1975 2000 high with older data, ranging between 0.904 and 0.975 (as is shown in Figure 2). At the Figure 2: Accuracy based on labelled test set of same time it can, perhaps unsurprisingly, be 3,873 unseen examples from the 20 decades included noticed that the accuracy (slightly) decreases in COHA (1810-2009). The classification accuracy as the linguistic data ages. ranges between 0.904 in the earliest decade, and 0.975
Overall, these accuracy scores are encour- in the most recent decade. aging, in that they highlight the efficiency of BERT-embeddings in identifying specific usage types of a grammatical construction, such as [BE about], in a large diachronic corpus. While that is an interesting observation in itself, the remainder of this paper will focus on assessing whether BERT-embeddings can be used to (semi-)automatically detect functional-semantic changes of grammatical constructions.
This paper’s methodological set-up starts from the assumption that changes in a word or
construction’s distribution, which can be captured in compressed numerical representations such
as BERT-embeddings, may indicate changes in its functional or semantic range. Following
earlier proposals using embeddings to detect lexical-semantic change in large diachronic
corpora [e.g. 11], this study will investigate whether known changes that have afected the [BE
about] construction can be detected by means of an embedding-based methodology combined
with Entropy Diference measures. More specifically, the task is conceptualized as follows: in
the case of lexical items, expansions or reductions in their distributional properties are often
equated with expansions or reductions of their possible interpretations – or, in other words,
with increases or decreases uncertainty regarding the exact interpretation of the lexical item.
To measure whether the “uncertainty over possible interpretations varies across time intervals”,
then, one can “compute the diference in entropy between the two usage type distributions in
these intervals” [
However, unlike the distributional changes that accompany lexical-semantic changes of
wellknown examples such as broadcast or gay, the distributional changes witnessed for [BE about]
– and many grammatical constructions like it – seem to proceed in a protracted sequence of
small steps, often spanning over several centuries [
The approach tested here works assumes that the researcher is interested in determining whether any of the usage types they distinguished has changed in the time span covered by their corpus with respect to a single reference point (for instance: Present-day data). For [BE about], the procedure involved fitting a logistic regression classifier on the embeddings created for the 952 present-day tokens, and applying it to a test set of 200 examples from each decade included in COHA (cf. Section 2.1). Subsequently, for each test token xi and each label y ∈ Y , the conditional probability p(y|xi) is computed to assess the uncertainty of the classifier in labelling the unseen examples. The resulting conditional probability over each label for each test token is then summarized in an entropy score, H:
H(xi) = − ∑ p(y|xi) ∗ log p(y|xi) y∈Y (1) If the entropy score changes over the 20 decades included in COHA, one could take this as an indication that the distributional properties of the test tokens in a particular category have shifted over time. Such shifts could be indicative of proper (subtle) functional-semantic change, or of increased or decreased use of the construction under scrutiny in diferent genres or text types. Conversely, if the distributional properties of a linguistic item or construction have not changed over the time span covered by the corpus, we should not expect to witness any changes in the certainty by which the model classifies tokens to its true usage type category (i.e. entropy).
Of course, it is difficult to assess the efectiveness of this method if we do not know whether [BE about] has actually undergone any functional-semantic changes in the period covered by COHA. As a means of assessment, then, I will first formulate the expected results for each of the construction’s three main usage types based on prior literature.
The first usage type under consideration is the futurate use of [BE about], illustrated in
examples (2)-(4). In present-day English reference grammars, the phrasal expression be about to
is commonly described as a so-called ’quasi-auxiliary’ which can be used to make a temporal
reference to the future [
Sheen, yes THAT Charlie Sheen, is about to become the best-paid actor in a comedy on TV.
(2006, COHA)
Just as I am about to step into the shower, the phone rings. It is George Stephanopoulos.
(2002, COHA)
But when he saw the sneer on St. Exeter’s face, Logan knew things were about to get much
worse. (2004, COHA)
From the relatively sparse number of accounts on the diachronic development of [BE about], it
can be concluded that the construction had already grammaticalized into a marker of
(immediate) future by the 19th century [
However, the [BE about] future did undergo a distributional change: in Present-day English, the phrase almost exclusively occurs with a to-infinitive complement clause, whereas 19th century and early 20th century texts also contain a variant with an ing-clause complement (as in (5)-(6)).
I really thought all my bones were disjointed, and that my soul was about taking a last farewell of my poor body. (1812, COHA) I was trying to sleep, and just as I was about succeeding Henderson called out: ‘[...]’. (1902, COHA).
If accurate, the method should detect that the distributional properties of futurate [BE about] have narrowed slightly over the course of the 19-20th century. 2.2.2. approximative about [BE about] often occurs as a marker of approximation. Note that the use of approximative about is not restricted to contexts with the verb be. The current sample therefore only represents a subset of the possible occurrences of approximative about, some examples of which are listed in (7)-(9):
When he was about 12, his parents left him and his siblings at an orphanage for five months. (2006, COHA) Though you hate to say anyone is recession-proof, U2 is about as close to that as you can get. (2009, COHA)
It was about at that time that Takemore disappeared from the township too. (2001, COHA)
With respect to its diachronic development, it has been shown that the spatial preposition
about approximative use of about – much like the near-synonymous use of around and various
other adpositions in other languages – developed into “approximative qualifiers of numerical
expressions and other amount expressions” [
In the third and final category, we find a group of what we could call ‘descriptive’ uses of [BE about], in which the phrase [BE about] can be roughly paraphrased as ‘regards’, ‘is (primarily) concerned with’. Its occurrences commonly involve clarifications of why situations are occurring (e.g. (10)), as well as descriptions of the theme, topic (or plot) of conversations, books, and iflms (e.g. ( 11)-(12)): (10) ... he wonders what the fighting is about and and who is fighting whom. Is it North against
South again? (2008, COHA) (11) The latest news is about Amanda! Haven’t you heard? (2000, COHA) (12) Mars and Venus Collide is not just about men understanding women. It is also about women understanding themselves and learning how to ask efectively for the support they need. (2008, COHA) Unlike with the previous two usage types, the number of historical and diachronic accounts that treat the descriptive use of [BE about] are not sparse, but virtually non-existent. Still, a quick scan of the dated examples listed in the Oxford English Dictionary suggests that the use of [BE (all) about] with an animate subject (e.g. a person, organization, or company) to describe what the subject is ‘primarily concerned with’ or ‘fond of’ constitutes a relatively recent (i.e. mid-20th century) phenomenon (e.g. (13)-(14)). (13) ... give him your authenticity spiel and how radio should be all about the music. (2009, COHA) (14) I’m all about the blindfold. There’s something intensely sensual about not knowing where you’re going (2005, COHA).
In other words, the types of elements that can occur in the subject slot in the descriptive subtype of [BE about] appear to have expanded during the time period captured by COHA, which may indicate that the descriptive use of [BE about] came to cover a broader semantic range. Given that the distributional properties of the descriptive use in the early 19th century were quite diferent from the present-day, we would again expect to find a shift classification uncertainty.
In sum, the (in)stability of the classification entropy should reflect the following: The distributional properties of futurate [BE about] have changed slightly during the 20th century. Having lost the ability to occur with ing-complements, it seems that the futurate use has narrowed. As with the futurate use, the distributional properties of the descriptive use have changed. More specifically, the descriptive use has expanded or broadened. By contrast, the distributional properties of the approximative use have remained stable.
As a starting point, it is worth considering real
the distributional range of the [BE about] con- 1.4 fadupetpsucrroreixptive
struction as a whole. It appears that entropy 1.2
1h.a0s9)in,daenedd, ianscrseuacshe,d iotvceorutlidmbee( fflargogmed0.a7s6 ato H1.0
case of semantic broadening [
What such a general test does not reveal is For each test token and each label, the conditional whether there have been any changes within probability has been computed to assess the uncerthese major usage types. For instance, there tainty of the classifier in labelling the unseen examis no immediate indication that anything has ples. The resulting conditional probability over each changed about the descriptive use besides its leanbterolpfyorsceoarceh, wtehsitchtogkreanduisalltyhednecsruemasmesarfiozredthien daenoverall frequency. scriptive use, and, to a lesser extent, for the futurate use.
To assess whether embeddings can be used to detect changes within the three major usage types, one could explore treating the semantic change detection task as a classification experiment. As explained in Section 2.2, the goal of the classification entropy test is to establish whether any of the usage categories of [BE about] has changed compared to a present-day reference point. We expect to find Entropy Diferences over time in two of the three usage types: the descriptive use, and, to a weaker extent, the futurate use. This expectation appears to be borne out (Figure 3).
A fair point that can be raised is that the attested decrease in uncertainty does not in fact signal that the usage of a construction has changed, but rather reflects a decrease in the extent to which embeddings created by unsupervised BERT (trained on predominantly Present-day English data) as the linguistic material more generally becomes older (and, consequently, less familiar). Still, the latter explanation loses at least some of its plausibility when considering the expected, relative stability of the approximative use. It is furthermore reassuring that the slight 80 60 40 20 0 20 40 60 2000 1975 1950 1925 1900 1875 1850 1825 (a) futurate (b) descriptive increase in classifier certainty appears to coincide with the decline of [BE about Ving], which renders futurate [BE about] more uniform and, consequently, more unambiguously recognizable.
The suggested shifts are also evident from the visualizations in Figure 4a and 4b, which
present what one could call a ‘time-sensitive t-SNE’ representation of the futurate and
descriptive use of [BE about]. If one wishes to avoid the use of an indirect, present-day reference
point to query a corpus for potential constructional changes, it would of course also be possible
to examine token groupings (as apparent from a time-sensitive version of t-SNE [
Using Figure 4, for example, one can examine the token groupings which are dated towards the beginning of the corpus (representing groupings of the [BE about Ving] pattern), or the smaller, markedly recent grouping top center left (containing solely negative uses, expressing absence of intent, e.g. Wang’s not about to forgive you (2007, COHA)). Note that the relative size of the time-specific token groupings discussed here may afect the extent to which summary statistics (such as the average pairwise distance between tokens or the silhouette score of clusters over time) capture their emergence or disappearance.
All in all, the present assessment of the use BERT embeddings and uncertainty measures to detect functional-semantic change in grammatical constructions seems largely positive. However, there are a number of potential pitfalls that must be addressed.
A first, smaller point that could be raised concerns the attested change in the [BE about] futurate. One may argue that this is, in fact, a formal rather than a functional-semantic change. What we witness is a reduction in the variability of (near-synonymous) complement clause types following the futurate, but this distributional change does not (straightforwardly) mark a shift in the futurate’s meaning. Still, what has been detected is of value to linguists, as the model has picked up a reduction in semasiological variation between a current and a currently obsolete complementation pattern.
Second, one may, as pointed out earlier, not be fully convinced that uncertainty measures determined by an unsupervised, present-day neural language model are the most reliable measure to detect semantic shift. First, it is unclear to what extent the fact that BERT has been pre-trained on present-day English material afects its performance with respect to the gradually aging data. Second, if tokens from a decade d are flagged as yielding a high degree of classifier uncertainty with respect to the reference point r, one should not be too quick to assume semantic change proper has taken place: in fact, this may be due to a diference in how the tokens are distributed across (sub)genres in d and r. In this study, I minimized both problems somewhat, but the concerns are still legitimate. With respect to genre variation, it helps to work with a carefully balanced corpus such as COHA (if available), or to try and incorporate meta-information on (sub)genre in the model [e.g. 26]. With respect to the possible ‘present-day bias’ of the pre-trained model, it is reassuring to see that attest stability with usage types that are not known to have changed, and that similar conclusions on possible distributional shifts can be arrived at by examining time-sensitive t-SNE plots.
However, it is important to stress that the explored approach solely considers uncertainty
with respect to the present-day reference point: while the classification entropy test successfully
pointed out that the descriptive use of [BE about] had undergone some changes, the decrease
in entropy cannot be equated to semantic narrowing. Instead, given that the descriptive
use of [BE about] rather seems to have shifted and broadened, the classification entropy test
merely indicates that the tokens have become more like the present-day examples and linguistic
material the model has been trained with. Second, the explored approach is limited in the
sense that the number (and nature) of usage types is imposed anachronistically to
non-presentday data. Since the procedure relies on a single reference point, it will not straightforwardly
lfag any usage types that are absent in the training set, and it may erroneously impose the
pre-defined category labels onto tokens representing obsolete usages. A further indication that
the method may be problematically biased towards present-day language can be found when
the model’s actual classification errors are considered in more detail. In the case of the [BE
about] futurate, the overall classification accuracy is remarkably high at 0.985, with only 22 of
the 1517 examples not being recognized as futurates. On closer inspection of those 22 mistakes,
it appears that 19 of them involve the now obsolete [BE about Ving] pattern. Given that there
are 102 examples of [BE about Ving] in the test set, this amounts to an error rate of 18.6%.
Furthermore, the mistakes are of the type illustrated in (15), where a Present-day descriptive
interpretation (i.e. Napoleon was fond of going to war with England) is erroneously imposed:
(15) Jeferson obtained the consent of Congress to make an efort to buy New Orleans and West
Florida, and sent Monroe to aid our minister in France in making the purchase. When the ofer
was made, Napoleon was about going to war with England, and, wanting money very much, he
in turn ofered to sell the whole province to the United States. (1897, COHA)
Because the use of data-driven, automated methods of semantic annotation and analysis is
appealing to researchers precisely because it could help avoid such anachronistic interpretations
of historical language [e.g. 29], it is of course unfortunate that they still occur at a reasonably
high rate. Yet, it should still be acknowledged that the very fact that word and phrase
embeddings created by BERT did succeed in recognizing diferent grammatical usage types in
Present-day language inspires hope that these problems can be tackled when models such as
these are trained on contemporary linguistic material and (following proposals such as [
I am grateful to Folgert Karsdorp for his advice on how to implement parts of the analysis.