In Information Retrieval and Natural Language Processing, representation of discrete objects, e.g., words, usually relies on embedding in vector space; this representation typically ignores sequential information. One instance of such sequential information is temporal evolution. For example, when discrete objects are words, their meaning may smoothly change over time. For this reason, previous works proposed dynamic word embeddings to model this sequential information in word representation explicitly. This paper introduces a representation that relies on sinusoidal functions to capture the sequential order of discrete objects in vector space.
However, such a embedding cannot deal with the spatially or temporally sequential information of objects. One spatial scenario is to encode word order in bag-of-words neural networks like Transformer [2, 3]. Regarding the temporal scenario, word meaning may change over time [4]. For instance, the word gay shifted from the meaning cheerful in the 1900s to the meaning frolicsome in the 1950s and finally to the meaning homosexuality since the 1990s [5].
In this work, we will focus on the temporally sequential aspect: temporal evolution. This work adopts sinusoidal functions to encode sequential evolution of word meaning change in vector space. The advantages over existing methods might be: 1) it is more eficient since the proposed method do not need to maintain a copy of word representation for each timestamp as required by previous works [6]; 2) it can be more efective to model semantic evolution since functions can deal with long-term but gradual meaning changes thanks to the continuity of
Binary coding: orders in 16 numbers (
A object-agnostic order (e.g., position and time) embedding [7, 2] is defined as: : R → R One may consider binary coding for order embedding. However, it is not diferentiable and thus unfriendly to neural networks. To this end, one may consider designing continuous coding with the same periodical property. Fig. 1 shows an alternative sinusoidal encoding [2, 8] with periods of 2, 4, 8, 16. Such continuity will facilitate back-propagation if such embedding is used in neural networks.
Object-aware dynamic evolution. Sequential encoding becomes more challenging when such sequential evolution is not shared among objects; for example, an individual word may change meaning over time, but other words may not share the same trend in meaning change. Therefore, such dynamic evolution processes are object-aware. Formally, evolution of an object with index can be formalized as a mapping from object (indexed in N) and time ( ∈ R) to a -dimensional vector:
: N × R → R
To smoothly model object-aware dynamic evolution, we represent each object as a continuous
function: a specific object embedding at time is represented as the values of the function when
the variable equals . More formally, our approach aims to learn a mapping that maps each
object to functions over time/order:
(
1The example is from https://kazemnejad.com/blog/transformer_architecture_positional_encoding Examples of are linear functions () = b + k with parameters b, k ∈ R or a sinusoidal functions () = b + v sin( + ) with parameters b, v, , ∈ R.
A typical way for word vectors is factoring positive point-wise mutual information (PPMI)
matrices [9]. Note that in a temporal scenario, PPMI matrices also changes over time. Assume
that the PPMI between a word pair (, ) at time is , (); our goal is to approximate , ()
by a dot product between dynamic word embedding of , denoted as ()() ∈ R, and a static
compass [10] of , denoted as ℎ() = vj ∈ R:
,() ≈ ()()ℎ()
(
⏟=1 =1 ⏟ ,⏞, ⏟ ,⏞, Therefore, , () is a weighted sum of sinusoidal functions plus a constant term Δ, i.e., , () = Δ + ∑︀=/12 ,, sin() + ,, cos() {,,}=/12 and {,,}=/12 are the co eficients and {Ω}2=1 are the corresponding frequencies. [11] states that linear combinations of sine and cosine functions could approximate all continuous functions in (). Thus, Eq. 7 could approximate any , () ∈ (), and therefore capture any word meaning evolution. Static object vectors, e.g., [12], can be considered as a special case of constant functions: = b, or a specific case of sinusoidal function when r = 0 or i is small enough. The additional parameters i and r are expected to capture the dynamic aspect of word meaning evolution. Intuitively, long periods reflect some long-range evolution, although in practice, such sinusoidal functions would not necessarily be periodical with an extremely long period in a limited timespan [13].
This paper proposes a sinusoidal parameterization to capture the sequential aspects of objects embedded in vector space. We focused on modeling change in word meaning over time; the considered parameterization is promising since, in principle, it could approximate any word meaning evolution. We are currently focusing on the evaluation of the proposed approach to investigate both its efectiveness and eficiency. Experiments will consider diverse tasks, e.g., temporal analogy [6] and semantic change detection [14]. Future work will consider other discrete objects, e.g., user profiles. Moreover, further theoretical and empirical investigations are needed to deal with the optimization issues when sinusoidal activation functions are used, i.e., infinity local minima [13]. The work is supported by the Quantum Access and Retrieval Theory (QUARTZ) project, which has received funding from the European Union‘s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 721321.