Word vector representations play a fundamental role in many NLP applications. Exploiting human-curated knowledge was proven to improve the quality of word embeddings and their performance on many downstream tasks. Retro tting is a simple and popular technique for re ning distributional word embeddings based on relations coming from a semantic lexicon. Inspired by this technique, we present two methods for incorporating knowledge into contextualized embeddings. We evaluate these methods with BERT embeddings on three biomedical datasets for relation extraction and one movie review dataset for sentiment analysis. We demonstrate that the retro tted vectors do not substantially impact the performance for these tasks, and conduct a qualitative analysis to provide further insights on this negative result.
The introduction of word embeddings was a breakthrough in NLP. Early
approaches based on the distributional hypothesis | words that appear in the
same context tend to be semantically similar | such as word2vec [
Although such systems can be trained on data speci c to the domain of interest, it is not yet clear how we can encode factual knowledge or impose constraints in the embeddings. Knowledge bases typically provide this type of Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). information, hence it is reasonable to exploit them in order to obtain more accurate and explainable embeddings.
Retro tting [
In this paper, we aim to extend retro tting to operate with contextualized word embeddings. More speci cally, we propose two di erent methods that, as in the original retro tting approach, make use of similarity relations between words in order to move the respective embeddings closer to each other in the latent space. The rst method combines the embedding of a given test sentence with the embeddings of sentences involving similar words in the training set, while the second method replaces a word in the test sentence by all possible similar words and combines the resulting embeddings. We evaluate the proposed methods with BERT embeddings on three biomedical datasets for a relation extraction task and one movie review dataset for sentiment analysis, and compare them with an oracle topline and two baselines (weighted majority vote and class posterior averaging). We show that both methods do not substantially impact the performance for this task, and conduct a qualitative analysis to provide further insights on this negative result.
The paper is organised as follows. We discuss related work in Section 2, and present the proposed methods in Section 3. We describe the experimental evaluation setup in Section 4, and we analyze the obtained results in Section 5. We provide conclusions and discuss future work in Section 6. 2
There have been several attempts to improve the quality of word embeddings by incorporating knowledge into the process. Two main categories of methods can be distinguished, which we refer to as joint or post-hoc.
Joint methods integrate knowledge by retraining the embedding model from
scratch using a modi ed training objective. For example, [
Post-hoc methods surpass these limitations, since knowledge is inserted in the
word embeddings after training, regardless of the model used to obtain them.
The most popular technique among these is retro tting [
By default, all of the above retro tting methods can only be applied to
distributional word embeddings, i.e., a single representation vector per word. When we
shift to contextualized embeddings, each word in the vocabulary can have a
different representation in each sentence. An attempt to retro t contextualized
embeddings coming from ELMo is presented in the Paraphrase-aware Retro tting
(PAR) [
As in the conventional retro tting approaches discussed in Section 2, we assume a vocabulary of words V = fw1; : : : ; wng and an ontology of semantic relations between words in V. We can then represent in the form of an undirected graph (V; E ), where nodes correspond to words in V and edges (wi; wj ) 2 E V V to semantic relations between nodes. Now, suppose that we have a contextualized word representation model M, along with a training corpus Dtrain on which it is ne-tuned and a test corpus Dtest on which it is evaluated for a particular task. 3.1
Method A The rst proposed embedding re nement method, which we refer to as Method A, combines the contextualized embedding of a given word in the test set with the contextualized embeddings of all occurrences of all similar words in the training set. Let qi 2 Rd be the contextualized embedding of word wi 2 V coming from M for a given test instance3. Let us further denote by Ji the set of words wj 3 For simplicity, qi does not have a superscript for the test sentence as we only process one test sentence at a time. which are adjacent to wi according to , and by Kj the set of training instances where wj occurs. Then we de ne q^jk 2 Rd to be the contextualized embedding computed for all occurrences of wj in Dtrain, as index by k 2 Kj . The index sets Ji and Kj vary dynamically for every word.
The goal is to learn a new embedding qi that it is close to qi and to adjacent nodes in under the L2 norm by minimizing
L(qi) = kqi qik2 + X X
bijkkqi j2Ji k2Kj q^jkk2 The weights bijk must naturally depend on the number of neighbours jJij of wi, and on the number of occurrences jKj j of each neighbor wj in Dtrain. In the following we de ne them as bijk = cij djk = jJ1ij jK1jj ; ; 2 [0; 1) where cij controls the contribution of each neighbour and djk controls the contribution of each of its occurrences. For example, = = 0 results in equal weights bijk = 1 for all occurrences, while = = 1 results in weights bijk that sum up to 1.
Equating to zero the derivative of L with respect to qi and expressing the Pk bijkq^jk in terms of the mean q^j of all q^jk results in the following update rule: qi = qi + Pj Pk bijkq^jk 1 + Pj Pk bijk = qi + jJij 1 + jJij
Pj jKj j1
Pj Kj1 q^j :
The retro tting operation therefore takes the form of a weighted average of the original embedding and the embeddings of all occurrences of all similar words in the training set.
L(qi) = kqi qik2 +
X bij kqi j2Ji q^j k2 (1) (2) (3) (4) 3.2
Method B The second proposed method, which we refer to as Method B, does not involve Dtrain at all. Instead, everything happens at test time. Again, we utilise M to obtain the embedding qi of word wi for a speci c sentence in Dtest. In addition, we derive one embedding q^j for every word wj which is adjacent to wi according to . To do so, we create a new sentence by replacing wi with wj in the test sentence, and repeat for every adjacent node of wi in . The objective is once more to learn a new vector qi that is close to both qi and all q^j under the L2 norm by minimizing Similarly to the above, we de ne the weights as bij = jJ1ij ; 2 [0; 1):
Equating to zero the derivative of L with respect to qi and expressing the Pj bij q^j in terms of the mean q^j of all q^j results in the following update rule: qi = qi + Pj bij q^j 1 + Pj bij = qi + jJij1 1 + jJij1 q^j :
Again, the retro tting operation takes the form of a weighted average of the original embedding and the embeddings of all neighbouring words.
The main di erence between the two methods lies in the way we exploit the information coming from the knowledge graph. Method A typically results in a large number of neighbouring vectors q^ik that contain noise, since the context around the corresponding words di ers from that of the test sentence in general. In contrast, Method B generates fewer neighbouring vectors q^j that share exactly the same context as the test sentence being processed. 4
In this section, we rst provide information with respect to the data, the semantic lexicons and the contextual word embedding model we used to evaluate the proposed retro tting methods. Then, we describe the experimental evaluation and we suggest three alternative strategies for comparison. 4.1
Data
We consider two tasks: relation extraction from biomedical data4 and sentiment
analysis of movie reviews. Two semantic verb lexicons are introduced in [
ChemProt is a manually annotated corpus of relations between drugs/ chemical compounds and genes/proteins mentions found in PubMed abstracts. The relations are categorized into ten classes from which only ve are used during evaluation. The task is to predict whether a pair of such entities is related or not, and if so, output the type of relation.
The DDI corpus aims in the development of systems that can automatically detect drug entities and drug-to-drug interactions in biomedical text. The corpus itself consists of texts from the DrugBank database and abstracts from the MedLine database. Annotations were provided by domain experts that classi ed drug-drug interactions into four DDI types.
i2b2 2010 corpus promotes the study of extraction/classi cation/relations of medical problems, tests, and treatments. The data consist of discharge summaries collected from Partners Healthcare, Beth Israel Deaconess Medical Center, and the University of Pittsburgh Medical Center, where relations of medical problems-treatments were grouped into eight classes. 4 The biomedical datasets are included in the Biomedical Language Understanding Evaluation (BLUE) benchmark, as well as the preprocessing codes for creating the training, development and test sets. 5 We refer to each di erent version of the verb lexicons simply by adding the number of the verb classes next to its name, e.g., annotated-34.
For the sentiment analysis task, we use the exact same semantic lexicons as
in [
SST-2 (Stanford Sentiment Treebank) [
BERT Architecture and Retro tting
There are di erent locations within the architecture of BERT, where retro
tting transformations can be applied. In general, the model consists of 12
Transformer blocks [
The motivation behind these choices is related to the complex architecture of the model. We hypothesize that the impact of any change into the embeddings would be more noticeable as we get closer to the output space, rather than in earlier layers of the model. Thus, we started experimenting at the pooling layer, which is the closest to the output space, but the results were not promising. Consequently, we moved one step back at the output layer of the last Transformer block, and further back to the same place of the preceding Transformer block.
In the retro tting equations (1) or (3), we initially considered as qi the embedding corresponding to the word token in the test sentence, but preliminary experiments showed that this did not have an impact on the nal performance. To verify this, we replaced the embeddings of these individual words with random numbers, or even zeroes. Both cases did not a ect the performance, indicating that the output classi er is not very much dependent on single word embeddings. Instead, we focus on the [CLS] token embedding which is a weighted linear average of all word embeddings in the test sentence, it is closer to the output space, and has a bigger impact on the nal result. All q^ij in (1) correspond to the activations of the word token in training sentences, whereas all q^j in (3) correspond to the activations of the [CLS] token in modi ed test sentences. 4.3
Technical Details
For the relation extraction task we chose BlueBERT [
Grid Search Optimization In order to nd a good set of values for the retro tting hyperparameters ; , we performed a grid search using the development sets. For the rst approach, we used both annotated and expanded clusters and we searched for ; 2 [0; 2] with a step of 0.2. We do not proceed on testing Method A for SST-2, as it turns out to be inferior to Method B. For the second approach, we use all four lexicons for sentiment analysis in conjunction with Bing Liu's sentiment lexicon (explained in Section in 4.1), while for relation extraction we only used the 34 and 50 classes of the annotated clusters6. Once again, we performed a grid search on the development sets where we searched for 2 [0; 2] with a step of 0.2. 4.5
Alternative Classi cation Strategies In order to assess the ability of our method to leverage the information in the lexicons, we augmented all datasets by adding all modi ed sentences that occur by replacing the underlying word with a neighbouring one, and compared with the following alternative strategies:
Topline: Always selecting the true class of a test sentence as the nal prediction, if it was predicted by at least one of the original or the modi ed sentences.
Weighted majority vote (WMJ): Picking the predicted class with the most occurrences as the nal prediction out of the original and the modi ed test sentences. Here, we assigned a weight of 1 to the original and a weight of jS1j ; 2 [0; 1] to each modi ed sentence, where jSj is the total number of sentences for the current test input. We experimentally noticed that choices of outside [0; 1] did not a ect the nal prediction.
Average probabilities (AVGP): Averaging the probabilities of the predicted classes for both the original and the modi ed test sentences, and taking the class with the maximum probability as the nal prediction. 6 This is due to the extensive amount of neighbouring verbs on the annotated-16 and the expanded clusters, which signi cantly increases the computational cost.
In this section we present the results obtained from the grid search, and conduct additional experiments that give more insight on the reasons why the proposed methods yield a similar performance to the baseline model. 5.1
Grid Search Experimental Results After nding the best performing set of hyperparameters amongst all combinations of lexicons, Transformer blocks, and positions that were tested on the development set, we evaluated the corresponding model on the test set. We report the performance for each dataset in terms of micro F1-score for relation extraction, and accuracy for sentiment analysis7. The results for both retro tting approaches are displayed in Table 1. At rst sight, both approaches seem to have no signi cant impact compared to the baseline performance. More speci cally, Method A results in a decrease of performance on all datasets, while Method B slightly improves it for ChemProt and SST-2. Furthermore, we notice that in many cases the alternative strategies we propose work better than our retro tting approaches. This suggests that i) the use of the lexicons is meaningful, but ii) we have not yet found the correct way of exploiting this knowledge. It is also worth highlighting the abrupt decrease in test performance on the i2b2-2010 for the AVGP method. We assume this is due to the model outputting di erent probabilities for each of the modi ed sentences. To con rm this, we compared with the score obtained from WMV for every 2 [0; 1] with a step of 0.1, and we observed that for low values of the weight the performance is signi cantly worse. This indicates that the original sentence is more important than the modi ed ones, implying in turn that we should assign a higher weight on it. However, in AVGP the averaging equally favours each class, and thus performs poorly. 5.2
Euclidean Distance Ranking of Retro tted Vectors
In order to understand in greater depth how our proposed methods change the
embeddings in space, let us focus on a single test case8 where the proportion of
disagreements between the baseline model and the test case model is statistically
signi cant (based on McNemar's test). This points out that both models behave
di erently, but on average they result in similar performance. To further analyse
how Method A a ects the embeddings in the latent space, we randomly select
5; 000 (out of 18; 014) test sentences where we apply our method, and we compute
the corresponding activation of the [CLS] token before and after retro tting.
Next, we compute the Euclidian distance between every retro tted vector and
every [CLS] vector before retro tting. This results in a 5000 5000 matrix, where
7 This is the standard choice of metrics for these tasks and datasets [
Next, we compute the counts that correspond to good, neutral and bad cases for every pair of original-neighbouring word. These will show on average if a neighbour is a good replacement or not for a given word. Then, using the McNemar's statistical test, we create three reduced versions, one for each semantic lexicon, by selecting a neighbour for a given word with a 10%, 50% and 90% con dence level9. The higher the con dence level the more certain we are about replacing a word by another one, but the smaller the lexicon becomes (and vice versa). Finally, we repeat the grid search optimisation (see Section 4.4) and present in Table 2 the results for the best settings. 9 We use the con dence level percentage as a subscript to denote the reduced lexicon, e.g., FrameNet90%.
Overall, there is some gain in performance compared to the baseline on the development set which is expected. For example, Method B reaches Topline performance for FrameNet10%, which suggests that retro tting in the sense of averaging embeddings can be meaningful. Moreover, we can see that the Topline performance is almost identical to that of the baseline model on the test data. This is due to the limited size of the reduced lexicons10. Ideally, if the dataset were bigger, we would have selected lexicons with higher con dence level that would also be large enough to improve over the baseline, i.e., the Topline score would signi cantly outperform the baseline. 6
In this paper, we proposed two approaches that extend the original retro tting technique to operate with contextualized embedding systems. More precisely, we incorporated external knowledge coming from semantic lexicons into BERT contextualized representations. After conducting a large-scale series of experiments on three biomedical datasets for relation extraction, and one movie review dataset for sentiment analysis, we observe that both approaches do not substantially a ect the performance on these downstream tasks. Our test results show that the lexicons can be a useful source of information to further improve the results. However, the current experimental setting did not make it viable. This is demonstrated in our qualitative study, where we show that when we improve the quality of the semantic lexicons by selecting only relevant neighbours for a given word, the resulting lexicons are not su ciently large to be able to generalize at test time. In the future, we plan to experiment with more ne-grained tasks where we are certain about the knowledge source, and where we would not need to heavily depend on word statistics to apply the proposed method. 10 For example FrameNet originally consists of 1700 words and 90140 relations, while its largest reduced version, FrameNet10%, has only 1 word and 5 relations.
This research was partially supported by the European Union's Horizon 2020 Research and Innovation Program under Grant Agreement No. 952215 TAILOR and by the Inria Project Lab \Hybrid Approaches for Interpretable AI" (HyAIAI). Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scienti c interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).