In the general perception of the NLP community, the new dynamic, context-sensitive, token-based embeddings from language models like BERT have replaced the older static, type-based embeddings like word2vec or fastText, due to their better performance. We can show that this is not the case for one area of applications for word embeddings: the abstract representation of the meaning of words in a corpus. This application is especially important for the Computational Humanities, for example in order to show the development of words or ideas. The main contribution of our papers are: 1) We ofer a systematic comparison between dynamic and static embeddings in respect to word similarity. 2) We test the best method to convert token embeddings to type embeddings. 3) We contribute new evaluation datasets for word similarity in German. The main goal of our contribution is to make an evidence-based argument that research on static embeddings, which basically stopped after 2019, should be continued not only because it needs less computing power and smaller corpora, but also because for this specific set of applications their performance is on par with that of dynamic embeddings.
wanted to be able to describe the loss of information and performance that results from using
static instead of dynamic embeddings. But our surprising preliminary results were confirmed
by a paper which was published as preprint during our work [
While token-based embeddings succeed in representation tasks, this is less clear for
abstraction, since these capabilities are not included in common benchmark tests [
Since the initial presentation of BERT, there were eforts to convert BERT’s contextualized
token-based embedding (that is, the output of the respective Transformer layers from a
pretrained BERT model under certain input sequences) into conventional static, decontextualized
type-based embeddings. We follow the terminology used in the survey by Rogers et al. [
Almost all approaches aggregate the token embeddings across multiple contexts into a single
type embedding in some way [
Similar to previous works, we only compute the embedding for a small selection of types relevant for our evaluation datasets. Thus, we already remark at this point that the generated type-based embedding is not “full” – in the sense that we assign only vectors to types that are present in our evaulation set, not to all types in the vocabulary, as is the case in static type-based embeddings. This has consequences on the type of tasks we can use to probe this small embedding, hence we were required to reformulate some tasks to account for this limit.
The techniques independently proposed by Wang et al. [
1Unfortunately, this terminology might be misleading. In particular, it should not be confused with the compression technique called “knowledge distillation” utilized in, e.g., DistilBERT.
Word embeddings are evaluated either intrinsic with their ability to solve the training objective
or extrinsic by measuring their performance on other NLP problems [
For word relatedness, pairs of words are given a score based on the perceived degree of their
connection, which are then compared to the corresponding distances in the vector space. For
the word similarity task specifically, the concept of relatedness is dependent on the degree of
synonymy [
Both relatedness and similarity can also be evaluated via the word choice task, where each
test instance consists of one focus word and multiple related or similar words in varying
(relative) degrees. Concerning word choice datasets, the synonym questions in the TOEFL exam
are the most prominent. One test instance consists of a cue word and four additional words,
where exactly one is a true synonym of the cue word. The distractors are usually related words
or words that could generally replace the cue word in a given context, but would change the
meaning of a sentence [
Lastly, the word analogy task can be used to probe specific relations in the vector space.
Given two related terms and a cue term, a target term has to be predicted analogous to the
relation of the given word pair. For word analogy datasets, the Google Analogies test set [
The usual implementation to test embeddings on this task uses linear vector arithmetic.
For example, given analogy “man is to woman as king (the cue term) is to queen (the target
term)”, we test if queen is the closest type in the vocabulary to king − man + woman. This
implementation builds upon the supposition that the underlying embedding exhibits linear
regularities among these analogies – in above example, that would be woman − man ≈ queen
− king, i.e. there is a prototypical “womanness”-ofset in the embedding. [
Since it was computationally infeasible for us to distill embeddings from BERT that have a comparable vocabulary size to those of static embeddings, we found that this setup becomes unreliable: Due to the smaller vocabulary, we heavily restrict the search space in these analogy tests, making the prompts appear easier to solve. Following the example, consider vector v = king − man + woman. Due to the small vocabulary, there are only few distractors in the neighborhood of v. Consequently, the vector queen most probably is the closest type from the vocabulary and the prompt is answered correctly, but this is not a cause of a structurization of the embedding space.
We train the type-based embeddings on the German OSCAR Corpus (Open Super-large
Crawled Aggregated coRpus [
As outlined in the introduction, we want to ensure that the diferent language models are
trained on the same or similar corpora. While training of type-based models on our chosen
corpora were feasible for us, it was impossible to pre-train a BERT model from scratch.
Therefore, we choose a pre-trained German model GBERTBase provided by Deepset in collaboration
with DBMDZ [
As the most popular evaluation datasets are in English, we constructed a comprehensive, German test suite consisting of multiple datasets which cover diferent aspects based on already existing evaluation data.3 The tasks covered are: word relatedness (WR), word similarity (WS), word choice (WC) and relation classification (RC). In addition, the data probes semantic knowledge such as synonyms and morphological knowledge, namely inflections and derivations. See Table 1 for an overview of all test datasets.
Word Relatedness/Similarity For WR, we used the re-evaluated translation of
WordSim353 (Schm280) as presented in [
WR and WS are judged via the Spearman’s rank correlation coefficient between the human annotated scores and the cosine distance of all word pairs.
Relation Classification As outlined above, we incorporate the concept of linearly structured word analogies into the test suite not in the usual way, but using relation classification. Instead of predicting a target word through a given cue word, RC tries to predict the relation type of a word pair by comparing their ofset with representative ofsets for each relation type. Specifically, we are given a collection of relations R1, R2, . . . , Rk where each Ri is a set of word pairs. We now interpret the relation Ri as a set of ofsets: for each word pair (a, b) ∈ Ri, we consider the vector vb − va. For the evaluation, we use a median-based 1-nearest-neighbor classification: We “train” the classifier by choosing the median of Ri as decision object. More precisely, we define vector ri = median({vb − va | (a, b) ∈ Ri}) as decision object of Ri. We then test this classifier on all pairs from all relations, thus check for each pair (a, b) from Ri wheter ri is in fact the closest decision object to vb −va (with respect to ℓ1-norm). We evaluate these predictions by the “macro” F1 score, i.e. the unweighted average of the F1 scores under each relation type, respectively.
While this setup allows for diferent aggregations resp. distance functions other than median resp. ℓ1-norm, we surprisingly found this choice to be more successful than other candidates (such as those based on cosine distance) among all examined embeddings.
For the RC data we made use of two German knowledge bases: the knowledge graph
GermaNet [
Word Choice Lastly, we included the WC task. Here, we used a translated version of
the TOEFL synonym questions [
Initially, we also wanted to explore world knowledge (i.e. named entities) captured by the embeddings, such as city-body of water or author-work relationships. However most named entities consist of multi-word expressions which are difficult to model via type or token based embeddings. We therefore removed all instances where a concept consisted of more than one token in the datasets described above.
Comparing embeddings distilled from BERT’s token-based embeddings with traditional static type-based embeddings requires us to examine diferent possibilities on how to perform this distillation. Therefore, we compare these possibilities by evaluating the resulting embeddings on the discussed evaluation dataset. Given these results, we decide on a single distillation procedure and compute a BERT embedding to compare its performance against static embeddings in Section 5.
In order to systematically evaluate the diferent methods to compute an embedding from
BERT’s token-based representations, we follow and extend the two-stage setup of Bommasani
et al. [
We extend this description by prepending a zeroth vectorization stage, in which we make explicit how to transform the outputs of BERT’s layers into a single subword vector. While
Bommasani et al. tacitly concatenate all outputs to form a subword vector, we also allow to selectively pick specific layer output(s) as subword vector, or a summation of the layers.
This setup also allows us to choose pooling functions f resp. context combination functions
g which are not defined component-wise. Hence, we examine a wider range of choices for
vectorizations, f and g than previously considered, e.g., in [
∑ norm(vi) , n i=1 that takes a set of vectors as input, normalizes each to unit length, calculates the mean on these normalized vectors, and normalizes this mean again. We motivate this aggregation function from the fact that meannnorm(v1, . . . , vn) is the unique vector on the unit hypersphere that maximizes the sum of cosine similarities with respect to each v1, . . . , vn. Thus, mean-norm could also be understood as a “cosine centroid”. In particular, mean-norm, medoids, and aggregations based on fractional distances were included in our experiments searching for suitable distillations. Table 2 shows the functions we examined. In total, we examine 17 possible vectorizations, 8 subword pooling functions and 6 context combination functions.
Intending to find best-performing distillations based on the above outlined general setup, we evaluate diferent choices for the “free parameters” of the distillation process as shown in Table 2.
For each word w to be embedded, we retrieve n = 100 sentences from the OSCAR corpus
as context for w, where w occurs in a sentence of maximum sequence length of 510. If w has
< n but at least one occurrence, we sampled all these occurrences. Types w that did not
occur in the OSCAR corpus were removed from the evaluation dataset. For each occurrence
in sentence s, we construct the input sequence by adding the [CLS] and [SEP] token. The
respective outputs of BERT on all layers form the input for the vectorization stage. This
method of generating input sequences by sampling sentences largely agrees with the methods
proposed by [
Due to the large number of possible distillations, it was computationally infeasible to construct embeddings under all distillations. Therefore, in a first experiment, we examined the quality of all considered distillations on a smaller evaluation dataset, which consists of three subsets of the MEN, the Wiktionary, the GermaNet, and the TOEFL dataset. Then, after restricting the set of potential distillations to promising ones, we perform the same experiment but with the full evaluation dataset on all five datasets. In both cases, the evaluation is performed as outlined in Section 3.3.
Also, since the scores in the respective tasks are reported in diferent metrics, we opt for standardizing the respective scores in each task when comparing some embeddings’ performances. Therefore, for one specific task, we consider the standardized score as the number of standard deviations away from the mean over all model’s scores in that task. We then can report the mean standardized score of some model taken over all considered tasks.
The first run of the experiment with all examined distillations on the smaller datasets gave
strong indication that centroid-based distillations lead to significantly better-performing
embeddings than those distillations that consist of a medoid-based poolings resp. aggregations.
In fact, among the 13 distillations with the highest mean standardized score, all twelve
distillations with centroid-based poolings and aggregations are present (Nopooling, mean, median,
mean-norm). Numerical values are presented in Table 8 in the appendix. (Top 13 distillations
are underlined.) With this experiment, we contributed insight on the performances on
diferent aggregation functions not previously considered in the literature. The results suggest the
interpretation that centroids – which represent a vector cluster by some synthetic aggregate –
generally lead to better results than medoids – which represent a cluster by some member of
that cluster. Also, in our distillation setup, fractional norms do not appear to give an
advantage, as opposed to research that indicate that fractional distance metrics could lead to better
clustering results in high-dimensional space, e.g. [
Therefore, we continue our evaluation of these twelve centroid-based parameter choices in the
next experiment on the full dataset. We observe that, under the restriction on centroid-based
poolings and aggregations, the choice of vectorization (i.e. layer) has a much higher influence
on the embedding’s performance than the actual choice of functions f and g. This supports the
general hypothesis that diferent layers capture diferent aspects of linguistic knowledge [
Again, we ranked the 17 × 4 × 3 (# vectorizations × # restricted poolings × # restricted vectorizations) analyzed embeddings with respect to their mean standardized score over all ifve tasks; cf. Table 3. This leads to our observation that those embeddings based on a sum-vectorization outperform any of the other embeddings; hence we suspect that a vertical summation resp. averaging over all layers can provide a robust vector representation for BERT’s tokens, capturing the summative linguistic knowledge of all layers in a reasonable fashion. Also, we suspect that the smaller dimensionality of the sum-vectorization might give the embeddings an advantage in comparison to the vectorization that concatenates all layers: the former has 768 dimensions, while the latter concatenates 12 Transformer outputs plus BERT’s input embedding, leading to 768 × 13 = 9968 dimensions.
To fix a single embedding for further comparison with static embeddings, we choose the embedding with the highest mean standardized score, which is the embedding based on the distillation with vectorization sum, the pooling f = median, and the aggregation g = median. The respective mean standardized score is highlighted bold in Table 3. Thus, when we now speak of BERT’s distilled embedding, then we explicitly mean this distillation (sum vectorization, median pooling and aggregation). Note that this embedding consists of 768 dimensions. Nevertheless, we explicitly remark that the small diferences in performances do not admit the claim that the chosen distillation is a universal method that would always perform best in any scenario. Also, we want to highlight that the previously untreated median as aggregation function appears to cause some improvement, especially in the pooling stage. Due to its robustness against outliers, we recommend to always examine this aggregation in distillations of any form that convert from token-based to type-based embeddings.
Before training the type-based embeddings models, we preprocessed the German OSCAR
Corpus to ensure that models are trained on the same version of the data. Preprocessing has been
done using the word tokenizer and sentence tokenizer of NLTK.5 Additionally, all punctuation
has been removed. We trained all models using the respective default parameters and used the
skip-gram model for all embeddings. We only adapted the number of dimensions and window
size to create additional embedding models for comparison. While the recommended window
size is 5, a window size of 2 has proven to be more efective in capturing semantic similarity
in embeddings [
As explained above, we evaluate how well the models represent diferent term relations with four tasks: word similarity and relatedness, word choice, and relation classification.
The first general observation looking at Figure 2 is that BERT’s distilled embedding (again, sum vectorization, median pooling and aggregation) does not perform significantly better, contrary to our expectations. In fact, the type-based embeddings seem to be capturing term relatedness and similarity even better than the token-based embeddings distilled from BERT 5Natural Language Toolkit (https://www.nltk.org/) 0.9 0.8 0.7 e r o c s 0.6 0.5 0.4 in most tasks: in WS, WR, and WC, FastText Dim768WS2 produces the best results (see Table 4) while in RC, BERT achieves the best results on the morphological relations only.
The WR and WS tasks (MEN, Schm280, SimLex999) paint a similar picture. Both hyperparameters, window size, and number of dimensions lead to a slight improvement when reduced and increased, respectively. Most notably, the similarity task benefits the most (about 0.05 absolute correlation improvement with both parameters adjusted for fastText, see Table 4) from altering these parameters. While BERT is on par with or even slightly outperformed by the 300-dimensional type-based embeddings in the relatedness task, it performs better in the similarity task. The higher dimensional vectors however can compare to BERT’s performance on the SimLex999 dataset. Overall, every model seems to struggle with the more narrowly defined WS task when compared to the WR task.
The WC task (Duden, TOEFL) also shows a clear trend: all type-based embeddings exceed BERT’s performance noticeably, by a 0.06 accuracy diference minimum ( Duden, Word2Vec Dim300) and 0.23 maximum (TOEFL, FastText Dim768WS2). Altering the parameters of the type-based embeddings, similariliy to the WR task, results in marginally better performing vectors.
BERT’s embeddings perform considerably better in the RC task when compared to the 300dimensional embeddings. However, a substantial gain from the dimensionality increase can also be observed with GermaNet as opposed to the other datasets, leading to both FfastText Dim768WS2 and Word2Vec Dim768 surpassing BERT’s performance by 0.06 and 0.08, correspondingly. While the same trend appears on the Wiktionary dataset, the classification of morphological relations by BERT’s embeddings still remains uncontested with an accuracy of 0.91. From a human perspective, the morphological relations are rather trivial (some examples are presented in Table 7 in the appendix); even from a computational point of view, lemmatizing or stemming the tails of these triples could in theory reliably predict the individual heads. This implies that generally, BERT can reproduce these kinds of simpler relations the best, while traditional models capture complex semantic associations more accurately. We separately explored the individual performances of all relations in GermaNet and Wiktionary and discovered that the higher F1 score of BERT mainly stems from the derivations, indicating that the word piece tokenization of BERT might facilitate its remarkable performance. Controlling for the dataset and relation size in a linear regression did not reveal a correlation between the amount of overlap and F1 however.
From these experiments we can conclude that for word similarity and term relatedness use cases, employing regular fastText embeddings, optionally increasing the number of dimensions, is sufficient. Using embeddings with the same number of dimensions as BERT results in the static embeddings taking the lead in the WS and RC task for semantic relations, specifically.
More so, there appears to be no clear trend on whether BERT’s distilled embedding is
generally better (or worse) than others models. In certain tasks, it performs particularly well (e.g.,
Wiktionary), and in others particularly bad (e.g., TOEFL). To give some statistical estimate on
the diference in performance between BERT and other models, we employ Bayesian
hierarchical correlated t-tests proposed by Benavoli et al. [
Table 5 gives the results on this inference. Most prominently, it estimates that on a future unseen dataset, FastText Dim768WS2 most certainly will outperform BERT’s distilled embedding by at least 0.03 absolute score points (P = 89.1 %). Even on the relatively weak Word2Vec Dim300, the hierarchical model predicts roughly equal probabilities for either BERT being better vs. Word2Vec Dim300 being better (by at least 0.03 absolute score points, 47.9 % vs. 51.8 %).
Nevertheless, this quantitative analysis also has limits due to the stochastic model presumed by the Bayesian hierarchical correlated t-test. The model assumes that the performance differences among the datasets (δ1, δ2, . . . , δnext) are i.i.d. and follow the same high-level Studentt-distribution t(µ 0, σ0, ν); thus, the model assumes that the considered datasets are in some way homogeneous. Though all our datasets are meant to examine word similarity, the distinct diferences in performance of the embedding types we observe (see fig. 2), indicate that these datasets represent diferent aspects of word similarity, which certain language models capture 6We want to thank the anonymous reviewer who brought the potential of the Bayesian hierarchical correlated t-test to our attention. better than others. Hence in our use case, we see the limits of the assumptions made by the stochastic model, and in this light, the results of the Bayesian hierarchical correlated t-tests need to be interpreted cautiously.
The most important result from our experiments is that a widespread assumption in NLP
and Computational Humanities is not true: a context-sensitive embedding like BERT is not
automatically better for all purposes. Static embeddings like fastText are at least on par if not
better if word embeddings are used as abstractions of semantic systems. But our results are
subject to some important limitations. For example, we can think of several ways which could
increase BERT’s capabilities to represent word similarity, which we haven’t explored:
• Modify the training objective for the pre-training phase, for example by adding a task
which influences how the model represents word similarity.
• Fine-tune the model on a task to improve the representation of word similarity, for
example predict the nearest neighbour based on existing similarity word lists.
• Replace wordpiece tokenization back to full word tokenization, which has been reported
to improve performances in some contexts. [
In order to understand how the performance diferences we observed between static and
dynamic embeddings relate to the performance gains which have been observed by stacking
embeddings from diferent sources [
Wiktionary SimLex999 Schm280
BERT is better than fastText. As discussed above the Wiktionary dataset consists mainly
of inflections, for example singular vs. plural, or derivations, for example masculine form of a
noun (‘Autor’) vs. female form (‘Autorin’). More examples are listed in Table 7. Maybe more
sophisticated approaches combining the diferent embeddings like [
Exploring the behaviour of the diferent embeddings we also came across a noticeable difference between the BERT-based embeddings and the static embeddings (see Figure 4). We calculated the distances between 956 synonym pairs, using synonyms as defined by GermaNet in one setting and defined by Duden in the other. To make the results comparable we standardized each of them by drawing 956 random word pairs and based our calculation of the mean distance and the standard deviation on them. Then we expressed the cosine distance of the synonyms in standard deviations away from the mean distance. The results show for both datasets a much larger spread for the static embeddings indicating that the BERT vectors occupy a smaller space, an efect which is not related to the dimensionality of its vectors. 0.15 y its0.10 n e d 0.05 15.0 12.5 10.0 7.5 5.0 2.5 model-standardized cosine distance
This seems to be in accordance with results from Ethayarajh [
To summarize, our main take away is not a recommendation for a specific static word embedding, rather we think it is worthwhile to continue research on static word embeddings – at least for researchers working in the field of Computational Literary Studies –, because their representational power as abstractions of semantic systems is on par to that of dynamic embeddings, the needed computing power is much less and the minimal size of the corpora needed to train them is also smaller. What we need in the field of Computational Literary Studies is a more robust understanding how the quality of embeddings is related to the size and structure of datasets, methods to improve the performance of static embeddings trained on even smaller datasets, maybe by combining them with knowledge bases, and more evaluation datasets for languages beyond English.
poolings ifrst last l1medoid l2medoid l0.5medoid l0.5medoid l1medoid l2medoid mean meannorm median l0.5medoid l1medoid l2medoid mean meannorm median l0.5medoid l1medoid l2medoid mean meannorm median l0.5medoid l1medoid l2medoid mean meannorm median l0.5medoid l1medoid l2medoid mean meannorm median nopooling l0.5medoid l1medoid l2medoid mean meannorm median mean l0.5medoid l1medoid l2medoid mean meannorm median meannorm l0.5medoid l1medoid l2medoid mean meannorm median median l0.5medoid l1medoid l2medoid mean meannorm median all all all inputemb L1
L2
L3
L4
L5
L6
L10
L11
L12
L1-4 L9-12 sum
all L7 L8 L9 vectorization/layer inputemb L1
L2
L3
L4
L5
L6
L10
L11
L12
L1-4 L9-12 sum all inputemb L1
L2
L3
L4
L5
L6
L10
L11
L12
L1-4 L9-12 sum
all L7 L8 L9 vectorization/layer L7 L8 L9 vectorization/layer L7 L8 L9 vectorization/layer L7 L8 L9 vectorization/layer L7 L8 L9 vectorization/layer L7 L8 L9 vectorization/layer inputemb L1
L2
L3
L4
L5
L6
L10
L11
L12
L1-4 L9-12 sum all inputemb L1
L2
L3
L4
L5
L6
L10
L11
L12
L1-4 L9-12 sum all
Germanet
Wiktionary
1
0
1
0
1
0
2
1
0
1
0
1
0
1
0
1
2
1
2
1
2
1
2
1
2
1
2
1
2
3
e
r
o
c
s
d
e
z
i
d
r
a
d
n
a
t
s
e
r
o
c
s
d
e
z
i
d
r
a
d
n
a
t
s
f=nopooling, g=mean
f=nopooling, g=meannorm
f=nopooling, g=median
f=mean, g=mean
f=mean, g=meannorm
f=mean, g=median
f=meannorm, g=mean
f=meannorm, g=meannorm
f=meannorm, g=median
f=median, g=mean
f=median, g=meannorm
f=median, g=median
f=nopooling, g=mean
f=nopooling, g=meannorm
f=nopooling, g=median
f=mean, g=mean
f=mean, g=meannorm
f=mean, g=median
f=meannorm, g=mean
f=meannorm, g=meannorm
f=meannorm, g=median
f=median, g=mean
f=median, g=meannorm
f=median, g=median
f=nopooling, g=mean
f=nopooling, g=meannorm
f=nopooling, g=median
f=mean, g=mean
f=mean, g=meannorm
f=mean, g=median
f=meannorm, g=mean
f=meannorm, g=meannorm
f=meannorm, g=median
f=median, g=mean
f=median, g=meannorm
f=median, g=median
f=nopooling, g=mean
f=nopooling, g=meannorm
f=nopooling, g=median
f=mean, g=mean
f=mean, g=meannorm
f=mean, g=median
f=meannorm, g=mean
f=meannorm, g=meannorm
f=meannorm, g=median
f=median, g=mean
f=median, g=meannorm
f=median, g=median
To compare two embeddings on our datasets, we have employed the Bayesian hierarchical
correlated t-test as described by Corani et al. [
As presented in Sec. 3.3, our tasks do not perform such cross-validation. Therefore, to adapt to the test, we modify our tasks as follows to obtain cross-validation results: • As the Relation Classification task ( Germanet, Wiktionary) is implemented as medianbased 1-nearest-neighbor classifier, it can be naturally extended to separate train and test sets. Given a train set of (labeled) word pairs and a test set of word pairs, we construct the decision objects (i.e., median) for the relations only on the training examples. Then we test the 1-nearest-neighbor classifier only on the test examples.
On each Relation Classification dataset, we perform 10 runs of 10-fold stratified cross validations to obtain 100 F1 scores. • For the Word Relatedness and Word Similarity tasks (SimLex999, Schm280, MEN), there is no natural way to implement a cross-validation, since these tasks measure correlation and are not “trained”.
Therefore, to mimic the 10-fold cross-validation, on each dataset we randomly sample 100 subsets that each contain 10 % of the respective dataset, and calculate the Spearman ρ on each of these subsets to obtain 100 correlation coefficients. • Similarly for the Word Choice tasks (Duden, TOEFL). We randomly sample 100 subsets containing 10 % of the respective dataset, and calculate the accuracies on each subset.
Fix a pair of two models we want to compare. For i-th dataset (of a total of q datasets), we calculate a vector xi = (xi1, xi2, . . . , xi100) of diferences of score on each cross-validation fold, using the same fold for each dataset. On these vectors x1, . . . , xq, we now can perform the Bayesian hierarchical correlated t-test using the Python package baycomp7 that implements the hierarchical stochastic model and performs the “hypothesis tests” that estimates the posterior distribution of the diference of score between the two models on a future unseen data set, as proposed by Corani et al.