The research described in this paper focused on exploring the domain of user pro ling, a nascent and contentious technology which has been steadily attracting increased interest from the research community as its potential for providing personalised digital services is realised. An extensive review of related literature revealed that limited research has been conducted into how temporal aspects of users can be captured using user pro ling techniques. This, coupled with the notable lack of research into the use of word embedding techniques to capture temporal variances in language, revealed an opportunity to extend the Random Indexing word embedding technique such that the interests of users could be modelled based on their use of language. To achieve this, this work concerned itself with extending an existing implementation of Temporal Random Indexing to model Twitter users across multiple granularities of time based on their use of language. The product of this is a novel technique for temporal user pro ling, where a set of vectors is used to describe the evolution of a Twitter user's interests over time through their use of language. The vectors produced were evaluated against a temporal implementation of another state-of-the-art word embedding technique, the Word2Vec Dynamic Independent Skip-gram model, where it was found that Temporal Random Indexing outperformed Word2Vec in the generation of temporal user pro les.
As of the time of writing, it is estimated that approximately 4.36 billion people of an estimated 7.6 billion globally are connected to the internet3. Some of the most successful of the 1.9 billion live websites in 2019 are social networking sites4, hosting Online Social Networking (OSN) platforms such as Twitter, Facebook and YouTube which allow users to connect digitally and share online content with each other. In order to capture, maintain and continuously increase the engagement of such a large user base in an incredibly complex global environment, the organisations behind these platforms are increasingly employing user pro ling tactics, where the preferences and interests of the user are modelled, clustered and learned in order to deliver tailored content directly to them in a scalable manner.
As described by Kanoje et al., [
The idea of temporally modelling or pro ling users can be motivated by
the observation that an individual user and their data are not static: Their
interests and preferences evolve and vary through time, often following patterns
such as trends, periodicities and spikes. This was demonstrated by
BonnevilleRoussy et al. [
It is clear that strong user pro ling techniques should be capable of capturing temporal variances in user characteristics. The idea of capturing temporal variances whilst modelling users and their interests through time has not been the subject of a great deal of research at the time of writing this document, despite the volume of research that exists in both user pro ling and NLP. Thus, exploring new viable approaches to capturing temporal variations in user pro les is the primary motivation for this research. 5 https://www.nytimes.com/2018/04/10/us/politics/
mark-zuckerberg-testimony.html 6 Word embeddings are a means of representing the semantic properties of a vocabulary of a corpus in a vector space, and are used widely in the area of NLP.
User pro ling using OSN data has been widely explored by the research
community, focusing on problems including personality inference and expertise
inference. Wald et al. [
When it comes to temporal user pro ling research, the quantity of research
conducted to-date is more limited. In their 2012 paper, Zhang et. al. [17]
proposed a user pro ling system which modelled users of a mobile network both
statically and dynamically using various di erent modelling techniques and
compared the performance using clustering algorithms. In 2017, Liang et. al. [
It is clear that understanding the temporal aspects of words and their semantics
is of major interest to fully understand the users of OSNs. Word embedding
techniques, as a group of widely used NLP techniques, are a strong candidate
to solve this problem. Temporality with respect to word embeddings considers
the way in which word semantics vary over time. There has been an increase in
interest in this variety of word embeddings, stemming from the fact that there is
now an abundance of time variant data sets available from major websites and
application platforms, as well as an increased appreciation of the fact that word
semantics do not remain static over time. Several methods have been proposed
in research which temporally extend static word embedding methods.
{ Jurgens and Keith proposed Temporal Random Indexing (TRI) [
algorithm generates word embeddings as a function of time, enabling
analysis and investigation into the evolution of word meanings over time. This
technique was used in their research for event detection in blog posts.
Subsequently, Basile et. al. [
As in the related literature regarding user pro ling, it was concluded by these researchers that temporal models tended to outperform their static counterparts. 3
It is clear from the research literature that there remain many opportunities to further investigate potential temporal user pro ling techniques, and that temporal word embedding techniques show a signi cant promise regarding understanding the temporal aspects of users interests based on their language usage. These observations motivated the decision to explore the application of Temporal Random Indexing to the problem of scalable temporal user pro ling using OSN data. 3.1
TRI is a word embedding technique, proposed by Jurgens and Keith [
For a given corpus C consisting of n documents where document d 2 C, a vocabulary V of m words can be extracted from C. Given this, the two steps involved in RI are as follows: 1. Assign a randomly generated Index Vector rp to each word wi in the vocabulary: wi 2 V . 2. Generate a semantic vector representation svi for each word wi, de ned as the sum of all random Index Vectors rp assigned to the words that co-occur with wi in a given context window given by the range j < p < +j for constant j. This is described by the following equation: svi = X
X
rp d2C j<p<+j
In the case of Temporal Random Indexing, before step 1 can be performed the corpus C must rst be annotated to contain metadata related to the creation time and date of the data. From this time data, the corpus C can then be split into k subsets C1, C2, ... Ck, where k is the number of time periods to analyse. To ensure that the Context Vectors produced by TRI are comparable across multiple time periods, the Index Vectors remain constant during the entire embedding generation process across each time period Tk, and the previous time period's Context Vectors are re-used as the initial state of the new time period's Context Vectors.
The major divergence between TRI and RI occurs in step 2 of the process. In RI, all data within the corpus is used to generate the vectors as time is considered as static and doesn't vary. In contrast, time is considered dynamic by TRI and thus a separate vector space is generated for each time period Tk contained within a document d. The equation governing this second process in TRI is given as follows: svi;Tk = X
X
rp d2C j<p<+j
Using this approach, it is possible to build vector spaces for each time period Tk over a given corpus Ck annotated with creation time metadata. Each word wi contained within the corpus has a unique Context Vector representation for each time period Tk considered, which are all built upon the same random Index Vectors. This allows for direct comparison between words within di erent time periods, since they are generated from a linear combination of the same random Index Vectors. 3.2
In this research, the previously described TRI technique is extended to capture a vectorised representation of Twitter users based on their language usage, using the same Index Vectors to generate a Context Vector i.e. a vector representing the user. Doing this allows for a direct comparison of not only words in the same vector space across multiple time periods, but also allows for the comparison of a single user with the shared vocabulary of all users in the dataset using vector mathematics.
This information about the user can only be captured by utilising the metadata related to the Tweets contained in the corpus: Speci cally, the creation time as well as the user id present in Tweet metadata. The additional step required to facilitate this involves augmenting the TRI technique to generate these useri vectors, achieved by collating the text written by a user in a given time period Tk and leveraging the same random Index Vectors used in generating the word embeddings.
Let Tk be a time period which spans between tkstart to tkend , where tkstart is a time that predates tkend . In order to build the user vector uvi for time period Tk, we consider all p words contained within the Tweets authored by user i within the time period Tk, where all Index Vectors, rp contained within the series of Tweets are summed as follows: uvi;Tk =
X rp p2Tk
Applying this additional step to the TRI technique, it is possible to construct a vector space for each time period Tk from a corpus C of n documents containing metadata related to Tweet creation time and user id. Each user ui has a distinct vector representation for a given time period in this vector space, given by uviTk , which is generated by accumulating the random Index Vectors for each time a word p was used by a useri in the given time period Tk. 4
The evaluation of this research focused on measuring how performant TRI is at modelling a user and their interests temporally based on their use of language. A temporal extension of the Word2Vec Skip-Gram model10 is used as a baseline comparison to the TRI user pro ling method described in this paper. The DISG model utilises the time period separated data, and independently generates embeddings for the words used in each time period. To generate a single user vector using DISG, the word vectors for each word used by a given user for a given time period are summed and averaged, resulting in a user vector being produced as the centroid of the word vectors used by that user in a given time period.
For the purposes of this research, the proposed system was evaluated by
following the same approach and collection of Tweets used by Liang et al. [
Gram (DISG) model, which is an inherently temporal model.
automatically generated ground truth is created by extracting all the hashtag
words associated with a speci c period of time. Speci cally, given the collection
of Tweets belonging to each one of the time periods, we extracted all of the
hashtags belonging to the Tweets in each time period, removing the hash character
('#') and lowercasing the hashtag word. Finally, we ranked all of the hashtags
by their tf idf , where the idf was computed on all of the Tweets authored by
a given user, and selected only the top 10 hashtags as ground truth. This is an
important di erence from the method described in [
By applying the techniques described in the Methods and Evaluation sections to the prepared Twitter dataset, and evaluating them against the generated ground truths using the trec eval evaluation tool, the results shown in Table 1 were observed.
TRI 0.0126 0.0142 0.0183
DISG
As can be observed from Table 1, although poor MAP and precision values were obtained for both models, TRI was found to outperform the Word2Vec DISG model for each of the time periods considered. This is a promising result for the usability of TRI and further work into improving the data processing, implementation and evaluation methods is likely to reveal further improvements in results.
This research has demonstrated the potential of the TRI technique as an e ective approach to temporal user pro ling through its application in analysis of temporal variations in language use.
The results obtained clearly demonstrate that the TRI technique outperforms Word2Vec, a word embedding technique currently considered to be state-ofthe-art in the NLP domain. This is a signi cant nding, and highlights the huge potential of applying word embedding techniques in temporal user pro ling applications. As a domain which has seen little research to-date, and given the ever-increasing volume of text content generated by users of some of the world's biggest OSN platforms, the growing importance of strong NLP techniques such as TRI cannot be overstated.
It is of the opinion of the authors that with further re nement and experimentation, a temporal user pro ling system employing TRI could be realised which is performant and scalable enough to be considered for use in online, web-scale environments.
Acknowledgement. This research was conducted with the nancial support of Science Foundation Ireland under Grant Agreement No. 13/RC/2106 at the ADAPT SFI Research Centre at Trinity College Dublin and Dublin City University and by the European Union's Horizon 2020 (EU2020) research and innovation programme under the Marie Skodowska-Curie grant agreement No.: EU2020 713567. The ADAPT SFI Centre for Digital Media Technology is funded by Science Foundation Ireland through the SFI Research Centres Programme and is co-funded under the European Regional Development Fund (ERDF) through Grant # 13/RC/2106. 17. Zhang, C., Masseglia, F., Zhang, X.: Modeling and clustering users with evolving pro les in usage streams. In: 2012 19th International Symposium on Temporal Representation and Reasoning. pp. 133{140 (Sep 2012). https://doi.org/10.1109/TIME.2012.16