Semantics in natural language processing is largely dependent on contextual relationships between words and entities in documents. The context of a word may evolve. For example, the word “apple” currently has two contexts - a fruit and a technology company. The changes in the context of entities in biomedical publications can help us understand the evolution of a disease and relevant scientific interventions. In this work, we present a new diffusion-based temporal word embedding model that can capture short and long-term changes in the semantics of biomedical entities. Our model captures how the context of each entity shifts over time. Existing dynamic word embeddings capture semantic evolution at a discrete/granular level, aiming to study how a language developed over a long period. Our approach provides smooth embeddings suitable for studying short as well as long-term changes. For the evaluation of the proposed model, we track the semantic evolution of entities in abstracts of biomedical publications. Our experiments demonstrate the superiority of the proposed model when compared to its state-of-the-art alternatives.
Word embeddings are low-dimensional vector space models
obtained by training a neural network using contextual
information from a large text corpus. There are several
variants of word embeddings with different features, such as
word2vec
To generate word embeddings, our framework trains a
model using a diffusion-mechanism for evolving concepts
within a scientific text corpus
Our approach uses the model for temporal highdimensional tf-idf representations introduced by (Camacho et al. 2018) to construct a training set. Construction of the training set is a one-time cost. The temporal highdimensional tf-idf representation (Camacho et al. 2018) is able to capture sudden short-term changes in the corpus. Additionally, it incorporates diffusion into the modeling to some extent by incorporating the time dimension smoothly. The framework presented by (Camacho et al. 2018) generates smooth tf-idf vectors for each word of a corpus at every timestamp, making it suitable for the generation of training data for our proposed model. One of the challenges of (Camacho et al. 2018) is that each word vector has a length equal to the number of documents in the corpus, which is not practical for analyzing a corpus containing thousands of scientific documents. Our goal is to construct a contextual low-dimensional temporal embedding space mimicking this high-dimensional representation without losing the essential temporal diffusion information encoded in the vectors. We introduce a neural-network-based framework that generates temporal word embeddings while optimizing for multiple key objectives. The temporal tf-idf representation from (Camacho et al. 2018) is used to obtain a baseline expected cosine distance (1.0 - cosine similarity) between pairs of word vectors at each timestamp. The expected cosine distance is used in the output layer of our proposed neural network. New low-dimensional embedding vectors – driven by a rigorous objective function to smoothly bring contextual entities close to each other – are generated in the hidden layer. The generated low-dimensional vectors are contextual and allow the discovery of latent (transitive) relationships that can’t be observed in the temporal tf-idf representation. For example, if words A and B are close to C, we expect words A and B to be close to each other. We further explain the objective function and the neural-network in Section 4.
The experimental results in Section 5 show that the
proposed method performs significantly better than the
stateof-the-art dynamic embedding models
2
Meanings of words in a language change over time
depending on their use
The state-of-the-art technique for language modeling is
word2vec, introduced by Mikolov et al.
A proposed solution to tracking semantic evolution is to
obtain a static representation for each timestamp in a
corpus, and then artificially couple these embeddings over time
using regression or similar methods
The drawbacks of using static word embedding models
to generate temporal representations have led to the
development of new techniques that can train the embeddings
for different timestamps jointly. The models use filters or
regularization terms to connect the embeddings over time.
Yao et al.
3
In this paper, we focus on timestamped text corpora, such as collections of scientific publications that have publication dates. Let D = fd1; d2; : : : ; djDjg be a corpus of jDj documents and W = fw1; w2; : : : ; wjWjg be the set of jWj noun phrases and entities extracted from the text corpus D. We consider each of the noun phrases and entities a word. Each document d contains words from the vocabulary (Wd W) in the same order as they appear in the original document of d. Every document d 2 D is labeled with a timestamp td 2 T , where T is the ordered set of timestamps.
The goal of this paper is to obtain a temporal word embedding model U from corpus D. Thus, for every timestamp t 2 T , we seek to obtain a vector representation uit for every word wi 2 W. The word embeddings U are represented as a 3-dimensional matrix of size jWj jT j juj where juj is a user-given parameter that indicates the size of a vector for a particular word at a particular time. We use the shorthand Ui to describe the 2-dimensional matrix of size jT j juj that represents word wi 2 W over time.
4
Each subsequent subsections below describes a major component of our objective function to generate diffusion-based temporal word embeddings. 4.1
We use the temporal tf-idf model (Camacho et al. 2018) to obtain high-dimensional time-reflective text representations of size jWj jT j jDj for training purpose. The vectors are formed using the temporal tf-idf weights of a word for every document in every timestamp. The temporal tf-idf weight of a word is computed using Eq. (1).
w^(w; d; td; t; &) = 0 where w^ is the weighted tf-idf value at timestamp t for the word w 2 W in document d 2 D, which was published at timestamp td. The term fw;d represents the term frequency of word w in document d, w is the number of documents that contain word w, and Wd is the set of words that appear in document d. The standard deviation of the Gaussian distribution function is represented by &, and is set by the user.
Next, we compute the cosine distance (1.0–cosine similarity) between every pair of words and store these as a distance matrix , where each element can be addressed as ijt 2 . This distance matrix becomes the training data for the expected distance between a particular pair of words (wi; wj ) 2 W at time t 2 T . We use the notation ij to represent a vector of size jT j with the temporal tf-idf-based cosine distance between (wi; wj ) 2 W for all the timestamps. The cosine distances are later used in the output layer of our proposed neural network.
One of our objectives is to obtain a low-dimensional word embedding model U such that computing the cosine distance between the word vectors results in a distance matrix that closely resembles . Equation (2) formulates this objective as #. In this case, we are optimizing the vectors in U to minimize the difference between the cosine distance of each pair of word vectors for every timestamp and the cosine distance from temporal tf-idf model in (Eq. (1)). The minimization of the difference will ensure that our model captures the same similarity as the temporal tf-idf model but ours will provide low-dimensional contextual vectors.
In this paper, the term dist(A; B) refers to the cosine distance between vector A and vector B. The cosine distance between two words vectors is bounded between [0; 1]. A cosine distance of 0 between two words vectors means that both words share the same context, while a cosine distance of 1 means that the vectors are completely orthogonal, thus does not share contextual similarities. The variable is introduced as a scaling factor to avoid numerical stability issues with values close to zero. The simplest form of our objective function is as follows.
jWj jWj jT j #1(U ) = X X X ( i=1 j=1 t=1 dist(uit; ujt) ijt) 2 (2)
In our work, we focus on the task of studying the semantic evolution of a word based on changes to its context. Thus, it is more important that our word embedding model correctly captures the relevant neighborhood of a word. Our experiments demonstrated that each word has a small number of relevant neighbors. That is, each word shares context with a small number of words. To take this into account in the objective function, we introduce a penalty when the temporal tf-idf-based cosine distance ijt is small, ensuring that our word embedding model captures the relevant context accurately.
jWj jWj jT j
#2(U ) = X X X
where is a scaling parameter to increase/decrease the
importance given to the samples with a smaller distance. Notice
that e ijt in Eq. (3) imposes a higher penalty to examples
with smaller baseline distances. The penalty is less when the
distance from the temporal tf-idf model is large. Equation (3)
supports the phenomenon that, for a specific word, most of
the words in the vocabulary are at a relatively large distance.
The large distances need not be a part of the penalty because
the objective function is only concerned about neighbors that
appear in the vicinity for the temporal tf-idf model.
Based on the diffusion theory
We use a Gaussian filter (Eq. (4)) to diffuse the contribution of each vector smoothly before and after the timestamp of the current sample. The filter uses a sliding window, going from the first to the last timestamp. is a user-settable parameter representing the standard deviation of the Gaussian distribution. A large value of means that the diffusion of word vectors is slow over time. A small standard deviation allows capturing short-term changes in meaning. (t; ) =
1 p 2 2 e (ti2 2t)2 with ti = 1; : : : ; jT j (4)
Equation (5) presents the updated objective #3 which includes the temporal diffusion of the word embeddings.
jWj jWj jT j #3(U ) = X X X The second important goal that our word embedding model should achieve is to be spatially smooth over time. Continuous or smooth temporal embeddings are those where the jWj jT j 1 "1a(U ) = X X
The main issue with this expression is that by forcing consecutive vectors to be very close together, we might be losing important information when the vectors drift apart in the original data. Thus, we introduce weights, !#, and !" to control the effect of each objective. The final objective function takes the form of Eq. (7).
Fa(U ) = #3(U )!# "1(U )!" An alternative form would be: or
Fb(U ) = !# log #3(U ) + !" log "1(U )
Fc(U ) = !##3(U ) + !""1(U ) 4.6
We implemented a neural network-based model using Tensorflow to generate our low-dimensional temporal word embeddings. An overall view of the architecture of our neural network is shown in Fig. 1. The goal of the neural network is to minimize Eq. (7). The embeddings for all words in all timestamps are generated in the hidden layer. We initialize the weights in the hidden layer in the range [0, 1]. The data used for training the model contains three inputs (one-hot encoding of a pair of words for which the cosine distance is known, and the timestamp) and one target value (cosine distance). The inputs are the indices for two random words wit and wjt, at timestamp t. The target value is the expected cosine distance between wit and wjt, obtained using the temporal tf-idf representations of Eq. (1).
5
We performed experiments using three different datasets: a synthetic dataset, PubMed Pandemic dataset, and PubMed COVID dataset. distance (e.g., Manhattan or Euclidean) between two vectors of the same word for consecutive timestamps is small. Equation (6) captures the expected behavior by penalizing significant spatial changes. (6) (7) (8) (9)
We generated the synthetic dataset consisting of 10,000 words and ten timestamps. For this dataset, we already know the 10-nearest neighbors of each word in every timestamp. Neighborhoods of larger sizes will contain random words starting at the 11th nearest neighbor.
stracts of pandemic and epidemic-related biomedical publications. The abstracts were published between years 2000 to 2020. We selected 3,000 most frequent biomedical entities for this dataset.
The PubMed COVID dataset contains 41,571 abstracts
of biomedical papers related to COVID-19, published in
2020. The corpus was collected from Kaggle COVID19
Open Research Dataset Challenge
We evaluate our temporal word embedding method by
comparing its performance with that of a regular tf-idf
model, the temporal tf-idf model (Camacho et al. 2018),
dynamic Bernoulli embeddings
We seek to answer the following questions. 1. What is the effect of introducing different penalty terms in our objective function? (Section 5.1) 2. How well do the models perform in terms of capturing the neighborhood of entities over time, compared to the temporal tf-idf? (Section 5.2) 3. How well do the models perform in terms of capturing changes in the neighborhood over time in the respective embedding spaces? (Section 5.3) 4. How well does our algorithm track the quick evolution of a specific entity, such as COVID, compared to other methods? (Section 5.4) 5. How well does our algorithm capture semantic evolution of a general term, such as pandemic, compared to other methods? (Section 5.5) 5.1
In this experiment, we study the effect of the different versions of our objective function on the quality of the temporal word embedding model, focusing on the task of tracking semantic evolution. The versions under this study correspond to #1 (2), #2 (3), #3 (5), Fa (7), Fb (8), and Fc (9). We quantify the quality of the resulting vectors with two different metrics: similarity and continuity.
The similarity is measured as the number of intersections between the word neighborhoods obtained using the temporal tf-idf model and each of the different versions of our objective function. The goal of the similarity evaluation is to quantify how well our model mimics the temporal tf-idf model. It must be noted that we did not expect to have a perfect match in the neighborhoods of words since the temporal tf-idf model representation does not take into account latent contextual relationships between words.
The continuity is measured using the average, maximum, and minimum mean squared errors (MSE) across consecutive timestamps for the word vectors obtained using the different versions of our objective function.
Fig. 2 shows the results for the similarity evaluation with the synthetic dataset described at the beginning of Section 5. The objective function labeled as Fa on the figure performs significantly better than the other formulations. If we discard Fb and Fc, it is possible to see how the similarity improves with the progression in which we developed our objective function. Furthermore, taking into account that only the top-10 nearest neighbors are known and set as accurate in the synthetic data and the rest of the neighbors are random, having an average of 8 intersections means that our a model can correctly capture the semantic evolution of the synthetic dataset.
Evaluating continuity is required to ensure that there is a smooth transition between timestamps for the vectors of the same word. A high average or minimum MSE value indicates that there is a significant movement of the word vectors over time in the embedding space. However, a small maximum MSE value would mean that the word embeddings are not following the trends observed in the temporal tf-idf model-based training. Thus, the best model is one that has high similarity with the temporal tf-idf model while maintaining a low MSE value.
Fig. 3 shows the results for the continuity evaluation. In this case, Fc has a continuity of 0.0, which, in conjunction with the similarity results, indicates that this objective function produces static, unusable vectors. The second smallest average MSE value is obtained with Fa, which also showed the best performance in terms of similarity. Thus, the final objective function is Fa (Eq. (7)), and we confirm that the smoothness penalty (Eq. (6)) has a positive effect both on the similarity and continuity results.
A major purpose of any temporal word modeling is to capture content similarity over time. We compare three models – TWEC, Bernoulli embeddings, and our temporal word embedding – with Temporal tf-idf (Camacho et al. 2018) in Fig. 4, using PubMed (pandemic) dataset. We use temporal tf-idf (Camacho et al. 2018) for this comparison because it models content smoothly over time. Each line in the figure represents average set-based Jaccard similarity between the 10-nearest neighbors of 1000 randomly selected entities using temporal tf-idf and the 10-nearest neighbors of the same entities using one of the three models. Fig. 4 demonstrates that our embedding model and TWEC have closer similarity with temporal tf-idf than Bernoulli embeddings. Additionally, our model has greater similarity with the neighborhood of temporal tf-idf in the earlier timestamps, compared to both TWEC and Bernoulli embeddings. Our model Averaged over 1000 randomly chosen entities from 328,908 PubMed (Pandemic) abstracts bernoulli temporal_embeddings temporal_tfidf tfidf twec smoothly spreads word-influence using diffusion over the years. As a result, our embedding model performs significantly better than other methods even when the vocabulary is smaller in the earlier timestamps. 5.3
An objective of a temporal embedding technique is to capture changes in the neighborhood of each word over time. The ability to capture changes allows us to study the evolution of concepts. This subsection provides an experiment to investigate how much change occurs from one year to another in the neighborhood using different models. We quantify change in terms of set-based Jaccard dissimilarity (1.0Jaccard similarity) between the neighborhood of a word in the current year and the neighborhood of the same word in the previous year. Average Jaccard dissimilarity over many words in a certain year for a model gives an overall idea of how much the model can detect changes in the neighborhood. Fig. 5 demonstrates average Jaccard dissimilarity (change) at each year for five different models – our temporal embedding model, Bernoulli embeddings, TWEC, and vanilla tf-idf computed independently at each year, and temporal tf-idf using 1000 randomly selected entities from the PubMed (pandemic) dataset. The plot shows that our temporal embedding model detects more changes in terms of average Jaccard dissimilarity compared to other models.
The Bernoulli embeddings capture the least amount of changes. Based on further investigation (not covered in this paper), we noticed that Bernoulli embeddings rarely capture any changes. These embeddings capture only a few longterm changes, whereas our temporal embedding model significantly captures both long-term and short-term changes. TWEC captures more changes than Bernoulli and temporal tf-idf, but lesser changes than the vanilla tf-idf. Our temporal word embedding performs even better than the vanilla tfidf. Contextual changes are best-captured using our temporal embedding because the objective function of our model spreads the effect of each word smoothly from the current year to other years. As a result, our model captures changes, in terms of average Jaccard dissimilarity, better than regular tf-idf and temporal tf-idf models.
Our model is clearly superior in terms of the ability to capture changes. In subsection 5.4, we explain how the superiority in the detection of changes in the neighborhood helps in analyzing evolving concepts, such as COVID-19. 5.4
In this experiment, we analyze the changes in the
neighborhood of the word COVID in the PubMed (COVID)
dataset. Fig. 6 presents how the similarities between the
entity COVID and some of its nearest neighbors–China,
epidemic, pandemic, and patients– change over time using (a)
TWEC model, (b) Bernoulli embeddings, (c) temporal tf-idf,
and (d) our temporal embedding model. The data contains
ranges of two-weeks from January to July of 2020. As we
already know, COVID-19 is, as of August 2020, considered
a pandemic – which is a global outbreak rather than a
local epidemic
Our temporal embedding (Fig. 6d) demonstrates that the word China had high similarity with COVID in the beginning. The similarity started to fall by the end of March. According to our model, starting at the end of march the word epidemic started to exhibit lesser similarity with COVID and the word pandemic started to show higher similarity. The temporal tf-idf model (Fig. 6c) demonstrates a similar trend. The trends match with our common knowledge regarding the COVID-19 pandemic. Also, TWEC (Fig. 6a) has an overall downward trend for the word China, but with zigzag movements over the timeline. Bernoulli embeddings (Fig. 6 b) do not demonstrate any change and capture a static similarity for the entire timeline. We noticed that the underlying vectors in Bernoulli embeddings change but the neighbors of a word do not change much.
We know that the number of COVID infected patients increased over the months of 2020. Our temporal embedding model (as well as the temporal tf-idf) captures the risingsimilarity of the word patients in the context of COVID quite smoothly (Fig. 6d). TWEC also has an upward trend which is less smooth. However, the Bernoulli embeddings do not demonstrate any changes in the similarity between the words patients and COVID.
This experiment demonstrates that our temporal embedding model captures the short-term changes in content (as shown by temporal tf-idf) while also capturing the context that we can track smoothly to study the evolution of a concept, such as COVID. In contrast, Bernoulli embeddings construct a context that is intractable in terms of similarity. TWEC provides noisy patterns that are difficult to interpret. Similar Words china epidemic pandemic patients -aJn411010 -aJn315120 -eF41b0130 -eF30b1540 -ra141050M -ra315160M -r14p1070A -r0315p80A -ya411090M -ya315110M -Jun411011 -Jun035121 l-Ju141013 l-Ju135114
Timestamps (ranges of dates in year 2020) (b) Bernoulli embeddings
Temporal Embedding Model
Similar Words china epidemic pandemic patients -aJn011014 -aJn025131 -eF030114b -eF045130b -ra500114M -ra061531M -r071014pA -r801530pA -ya090114M -ya051131M -Jun110114 -Jun215130 l-Ju130114 l-Ju141531
Timestamps (ranges of dates in year 2020) With the rise of the COVID-19 pandemic, it has become essential to study how biomedical scientists have dealt with a pandemic in the past years. Such an analysis requires a model that can capture long term changes. In this experiment, we attempt to track the closest term to the word pandemic in each year of the PubMed (pandemic) dataset, which spans biomedical abstracts from 2000 to 2020.
Each line of Fig. 7 plots the similarity of the top
nearestneighbor of the word pandemic in each year. The five lines
represent similarities using five different models – Bernoulli
embeddings, our temporal embedding model, temporal
tfidf, vanilla tf-idf, and TWEC. Notice that our temporal
embedding model demonstrates peak similarities in 2009/2010
and in 2020, when H1N1 influenza (swine flu) and
COVID19, respectively became prominent. This signal from our
temporal embedding model reflects the fact that the worst
pandemics in the last 20 years are the H1N1 influenza in
2009 (Sullivan et al. 2010) and COVID-19 in 2020
TWEC captures influenza and H1N1 in the middle of
the timeline but fails to capture COVID-related keywords
in 2020 as the closest entity to pandemic. In Fig. 7, the
Bernoulli model can pick up coronavirus as the nearest
neighbor of pandemic but it was not able to pick up influenza
in its trend. Moreover, coronovirus appears in all the years
as the top nearest neighbor of pandemic which is not
correct because the fact is that the coronavirus spread started in
2019 and became a pandemic in 2020
Based on the experiment presented in this subsection, our temporal embedding model has the ability to separate highly contextual words (such as H1N1 and COVID) of a concept (such as pandemic) via similarity-peaks. Our model helps in determining prominent neighbors of a concept in the past. bernoulli 1.1 1.0 coronavirus
smoking d an 0.9 " c i edm ro0.8 an hb0.7
g "p ie een tn0.6
s tebw reea0.5 0.1 0.0 africa 0200 1200 2200 3200 4200 5200 6200 7200 8200 900 001 1012 2012 3012 4201 5201 6201 7201 8201 9201 0202 2 2
TTimimessttaammpps
Our vectors are able to construct a peak for a prominent nearest neighbor because our method models diffusion. That is, a concept that appears today affects the past and the future to some extent, regardless of whether the concept directly appears in the contents or not.
6
This paper introduces a new technique to generate lowdimensional temporal word embeddings for timestamped scientific documents. We compare our model with existing temporal word embeddings. Our method generates a representation that: (1) can track changes observed within a short period, (2) provides a smooth evolution of the word vectors over a continuous temporal vector space, (3) uses the concept of diffusion to capture trends better than the existing models, and (4) is low-dimensional. Unlike previous models, our proposed model creates a homogeneous space over every timestamp of the embeddings. As a result, the generated vectors over timestamps can be used for prediction using conventional algorithms for predicting signals. Extrapolation of the embedding vectors to forecast a future neighborhood of a scientific concept is a future direction of this work.