=Paper=
{{Paper
|id=Vol-2573/paper1
|storemode=property
|title=Privacy-Aware Personalized Entity Representations for Improved User Understanding
|pdfUrl=https://ceur-ws.org/Vol-2573/PrivateNLP_Paper1.pdf
|volume=Vol-2573
|authors=Levi Melnick,Hussein Elmessilhy,Vassilis Polychronopoulos,Gilsinia Lopez,Yuancheng Tu,Omar Zia Khan,Ye-Yi Wang,Chris Quirk
|dblpUrl=https://dblp.org/rec/conf/wsdm/MelnickEPLTKWQ20
}}
==Privacy-Aware Personalized Entity Representations for Improved User Understanding==
https://ceur-ws.org/Vol-2573/PrivateNLP_Paper1.pdf
Privacy-Aware Personalized Entity Representations for
Improved User Understanding
Levi Melnick Hussein Elmessilhy Vassilis Polychronopoulos Gilsinia Lopez
Yuancheng Tu Omar Zia Khan Ye-Yi Wang Chris Quirk
Microsoft
{lemeln,huahme,vapolych,gilopez,yuantu,omkhan,yeyiwang,chrisq}@microsoft.com
ABSTRACT access-controlled documents that are not publicly available. Our
Representation learning has transformed the field of machine learn- goals are to help each user find, classify, and act upon these grow-
ing. Advances like ImageNet, word2vec, and BERT demonstrate the ing information stores, then to acquire and organize information,
power of pre-trained representations to accelerate model training. including facts and relationships among these entities. A crucial en-
The effectiveness of these techniques derives from their ability to abling step is to build reusable representations of this information.
represent words, sentences, and images in context. Other entity Most representation learning uses large, publicly-available docu-
types, such as people and topics, are crucial sources of context in ment stores to build generic embeddings. We believe there is also
enterprise use-cases, including organization, recommendation, and great value in user-conditioned representations: representations of
discovery of vast streams of information. But learning represen- phrases and contacts for each user learned on the information
tations for these entities from private data aggregated across user uniquely available to that user. First, building user-conditioned rep-
shards carries the risk of privacy breaches. Personalizing represen- resentations provides a huge amount of context. Often when there
tations by conditioning them on a single user’s content eliminates are ambiguous or overloaded concepts, the key people surrounding
privacy risks while providing a rich source of context that can their usage can disambiguate. Furthermore, a given user may extend
change the interpretation of words, people, documents, groups, the meanings of a given concept as they document and communi-
and other entities commonly encountered in workplace data. In cate new ideas. Perhaps most importantly, training a model based
this paper, we explore methods that embed user-conditioned repre- on only the communications and documents available to a given
sentations of people, key phrases, and emails into a shared vector user provides a clear and intuitive notion of privacy. Whenever we
space based on an individual user’s emails. We evaluate these rep- train on data beyond any user’s normal visibility, there is some po-
resentations on a suite of representative communication inference tential for capturing and surfacing information outside their view.
tasks using both a public email repository and live user data from Differential Privacy helps limit the exposure of any individual user,
an enterprise. We demonstrate that our privacy-preserving light- but preventing leakage across groups is more difficult. For instance,
weight unsupervised representations rival supervised approaches. certain privileged information may be discussed heavily by many
When used to augment supervised approaches, these representa- members of an administrative board, yet this information should
tions are competitive with deep-learned multi-task models based not be shared broadly across the whole organization. When training
on pre-trained representations. a user’s model on only data that that user can see, the possibility
for leaking information is removed. From the perspectives of both
leveraging a crucial signal as well as maintaining user privacy and
1 INTRODUCTION trust, user-conditioned representations hold great promise.
User-conditioned learning comes at a cost. Data density de-
Pre-trained embeddings are a crucial technique in machine learn- creases dramatically. State-of-the-art deep learned representations
ing applications, especially when task-specific training data is typically train on billions of tokens [12], whereas an individual
scarce. For instance, groundbreaking work in image captioning user’s inbox may only have a few thousand emails. Thus, we ex-
was enabled by reusing the penultimate layer of an object recog- plore shallower personalized approaches with lower sample com-
nition system to summarize the content of an image[24]. More plexity (though shallow models can be mixed with deep generic
recently, contextualized embeddings are setting the state-of-the-art models for empirical gains [10]). Furthermore, training must be
in a range of natural language processing tasks [12]. Training mod- performed for every user separately within the organization; in our
els to extract reusable representations from data is now an obvious case, this entails separate training runs for hundreds of millions of
investment. The next key research question is which context to users. Because the information available to the user is constantly
leverage. changing, maintaining fresh representations is also a challenge.
Our research is situated in the area of User Understanding: or- As computation and storage become cheaper, the overhead of
ganizing the information, documents, and communications that maintaining user-conditioned models is tractable only if the models
are available to each user within an organization. Users now com- are light-weight. Furthermore, we focus on task-agnostic repre-
monly retain huge mailboxes of written communication; members sentations that benefit a range of scenarios, amortizing the cost of
of larger organizations also have access to large repositories of computation. Finally, using models trained only on one user’s data
benefits privacy, which is an increasing concern for organizations
PrivateNLP 2020, Feb 7, 2020, Houston, Texas and individuals.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).
PrivateNLP 2020, Feb 7, 2020, Houston, Texas Melnick, Elmessilhy, Polychronopoulos, Lopez, Tu, Khan, Wang, and Quirk
1.1 Contributions previously favored items [1]. Our entity representations also embed
We present the first efforts in building per-user representations: phrases with contacts, but they are task-agnostic.
content-based representations that embed disparate entities, includ- Personalized language models have shown benefits in speech
ing contacts and key phrases, into a common vector space. These recognition [26], dialog [23], information retrieval [38], and col-
entity representations are different for each user: the same key laborative filtering [19]. These approaches model the user, but the
phrase and contact may have very different representations across representations are not generated on a per-user level; instead, all
two users depending on their context. We focus on slow-changing users share the same representation for an item or phrase. In our
entities like contacts and key phrases to minimize the impact of approach, the same item (phrase or contact) will have a different
delayed retraining: although one’s impression of their collaborators representation according to each user, since the entity representa-
may shift over years, months, or perhaps weeks, a representation tions are generated per user only considering the data available to
that is a few days old is still useful. To embed rapidly arriving that user. Nawaz et al. [29] present a technique to perform social
and changing items such as documents and emails, we present ap- network analysis to identify similar communities of contacts within
proaches that assemble representations of rapidly changing entities a user’s own data but their approach is limited to a single task. Yoo
from their content, including related contacts and key phrases. et al. [41] describe an approach for obtaining representations of
We evaluate these representations on a range of downstream emails using personalized network connectivity features and con-
tasks, including action-prediction and content-prediction. Simple, tacts. Their representations are used as inputs to machine learning
unsupervised approaches, especially non-negative matrix factoriza- models to predict the importance of emails. In our approach, key
tion (NMF) [25], produce substantial improvements in accuracy, out- phrases, contacts, and emails are all embedded in the same space.
performing task-specific and multi-task neural network approaches. Thus, we can use the task-agnostic representations for a variety of
We compare user-conditioned representations to representations tasks and obtain similarity scores for any pair of entities.
learned at the organization-level, where data from multiple users Starspace [39] introduces a neural-based embedding method that
are combined together into a larger undifferentiated store. User- maps different types of entities (graphs, words, sentences, docu-
conditioned representations mostly outperform the organization- ments, etc.) to the same space. While the entities are embedded
level approaches, despite decreased data density, presumably be- in the same space, like our work, their training uses global infor-
cause the additional context provides helpful signals to models. Fur- mation, and the loss function on which the network is trained is
thermore, user-conditioned representations sidestep issues related task-dependent. Our approach allows reusing the same representa-
to privacy preservation by not mixing data across user mailboxes. tions obtained using local data across a variety of tasks.
Our evaluation tasks bear some similarity to the evaluation of
knowledge base completion through embeddings of text and en-
2 RELATED WORK tities [36]. However, our entities are not curated knowledge base
Email mining [35] has been widely studied from different angles entities; they are phrases and people known to a particular user.
both for content and action classification. Spam detection has re- In query expansion, locally trained embeddings can outperform
ceived considerable attention both from a content identification global or pre-trained embeddings [13, 32] by incorporating more
and filtering view [8] as well as from a process perspective [16]. relevant local context. We exploit a similar insight in training only
Folder prediction is another task that can help better organize in- on the user’s own data and directly incorporating her context. Amer
coming emails [22]. Email content has also been used for social et al. [3] present an approach that trains a per-user representation,
graph analysis to learn associations between people, both for re- though their per-user embeddings perform worse than global or
cipient prediction [4] and sender prediction [17]. Action prediction pre-trained embeddings. In this paper, we demonstrate methods to
tasks that have been considered in the context of emails include train per-user representations that can not only outperform global
reply prediction [40], attachment prediction [15, 37], and generic representations but also generate them in a task-agnostic manner.
email action prediction [6]. In this paper, we use recipient pre-
diction, sender prediction, and reply prediction as representative
tasks to evaluate the quality of our learned representations. All 3 ENTITY REPRESENTATIONS
prior work on these and similar tasks has relied on per-task feature
We developed representations for three entity types: key phrases,
engineering and supervised model training. We show how entity
contacts, and emails. Key phrases (typically noun phrases) [18] can
representations generated in a task-agnostic manner can be used
appear anywhere in the body or subject of an email. Restricting
both in an unsupervised and in a supervised setting for these tasks.
to extracted key phrases limits the total number of entities for
Entity representations have been used extensively to provide
which representations must be learned. By “contacts,” we refer
personalized recommendations. Most such models build global rep-
to individual email addresses that appear in the From, To, or CC
resentations of users and items in the same latent space and then
fields of an email. For slow-changing entities like key phrases and
determine the similarity between the user and item through cosine
contacts, we can periodically regenerate a stored representation. We
similarity. The user embeddings can be built in a collaborative fil-
represent fast-changing entities such as emails with light-weight
tering setting by leveraging a user’s past actions such as clicks [30],
compositions of pre-trained entity embeddings (contact and key
structured attributes items interacted with, ratings offered on past
phrase embeddings) to minimize computational expense.
items [5], or even past search queries [2]. An extension to such
Since the median inbox size will be small (in both our data sets it
approaches is to combine embeddings of words or phrases with
is around seven thousand emails) there is not enough data per user
other types of data [33], such as embeddings of users [27] or their
Privacy-Aware Personalized Entity Representations for Improved User Understanding PrivateNLP 2020, Feb 7, 2020, Houston, Texas
Words + contacts + key phrases
to train useful deep learned representations. Indeed, our early at- the item … john@doe.com
Emails
tempts to train user-conditioned word2vec [28] embeddings yielded mail1 4 1 1
poor results and are not reported here. Therefore we only consid- Original corpus:
mail2 2 0TF-IDF matrix 0
ered approaches that were likely to perform well at low data density.
… ⋱
Aggregate information
3.1 Key Phrase and Contact Representations mailn 8 0 1 involving all contacts
and key phrases
We compute unsupervised entity representations for contacts and
key phrases by associating them with concatenated documents.
Contacts + key phrases
the item … john@doe.com
These concatenated documents are assembled from the user’s orig- x@y.com 0.1 0.3 0.25
inal documents; the emails in our experiments have From, To, CC,
a@b.net 0.2 0 0
Body, and Subject fields. Given a particular entity e, its concate-
… ⋱
nated document de is the concatenation of every email m in a user’s
inbox such that e appears in any of those fields. This concatenation due date 0.3 0 0.30
is done on each field f independently: every new concatenated Reduce dimensionality of
document de will have a corresponding field de, f for each field f concatenated document matrix
É
in the original document. We use to denote concatenation.
Ê
de, f = m f , where Me = {m : ∃f such that e ∈ m f } (1) Words + contacts + key phrases
Contacts + key phrases
m ∈M e
H
Stop words on the Scikit-learn stopword list are removed, as are
terms that appear in more than 30% or less than 0.25% of training
W
emails. We then generate a sparse numerical entity-by-term matrix,
using TF-IDF for most methods or just TF matrix in the case of LDA.
Initially one matrix is computed for each field f using the relevant Entity Compute Pseudo-inverse: H+
Representation Map from documents
portion of the concatenated document collection D f = {de, f }e . into embedding space
Each matrix is scaled according to a weighting factor w f to balance
its contributions, and finally these matrices are concatenated to
form a single matrix T .
Êp Õ Figure 1: Process of creating concatenated documents to represent
T = w f · term-matrix(D f ), where wf = 1 (2) contacts and key phrases given the count matrix from an email cor-
f f pus. We also demonstrate how this matrix can be factorized into a
low rank approximation to encourage inference over sparse items.
The weights of the different email fields are treated as hyperpa- The left matrix W can be interpreted as entity representations. Fur-
rameters and tuned empirically to perform well on the evaluation thermore, we can derive a mapping from the words, phrases, and
tasks. We found that weights of 0.4, 0.3, 0.2, 0.05, and 0.05 for the contacts in an email into this representation space using the pseu-
Body, Subject, From, To, and CC fields worked well. The rows of T doinverse H + of the right matrix H . Other low rank approximation
are sparse representations of entities – a simple and safe baseline. and composition approaches are explored as well.
We explored LDA, LSA, and NMF as a means of encouraging softer
matching through dimensionality reduction. constraining the low-rank matrices to be positive and adding a reg-
3.1.1 TF-IDF. Our baseline representation technique is sparse ularization term [25]. Specifically, given an input matrix T ∈ Rm×n ,
unigram TF-IDF vectors produced from the concatenated docu- we try to find matrices W ∈ Rm×d and H ∈ Rd ×n to minimize
ments.
3.1.2 Latent Dirichlet Allocation (LDA). Latent topic models us- |T − W H | + λ (|W | + |H |) (3)
ing LDA [7] over the term frequency matrix of the concatenated
documents (not the TF-IDF matrix) learn a mixture of topics for where λ is a regularization weight and | · | is the Frobenius norm.
each document. These learned vectors can act as entity represen- The W matrix serves as a representation for the entities. We ef-
tations. We can vary the number of latent topics to determine the ficiently compute NMF through the Hierarchical Least Squares
dimensionality of the resulting embeddings. algorithm [21].
3.1.3 Latent Semantic Analysis (LSA). A classic method for re-
ducing sparsity, LSA [11] builds a low-rank approximation of a TF- 3.2 Email Representations
IDF matrix T using the singular value decomposition: T = U ΣV T . The vocabulary of key phrases and contacts in one’s mailbox is
3.1.4 Non-negative Matrix Factorization (NMF). The SVD re- likely to grow slowly, and their meanings and relationships will also
construction has a few problems: the values of the matrix may be evolve gradually. By comparison, many new emails arrive every
positive or negative, and there is no explicit regularization term. day, so the “vocabulary” of email entities is constantly increasing.
Together, these issues may lead to strange or divergent weights, Thus, while it is possible to train representations for email in the
especially when the data is difficult to model with lower rank. same way that we do for key phrases and contacts, updating email
Non-negative matrix factorization (NMF) addresses these issues by representations on an ongoing basis would imply vast storage and
PrivateNLP 2020, Feb 7, 2020, Houston, Texas Melnick, Elmessilhy, Polychronopoulos, Lopez, Tu, Khan, Wang, and Quirk
From: terrie.james@enron.com |V|
Subject: Enron Prize Replied to: False 300
To: kenneth.lay@enron.com 100
Cc: cindy.olson@enron.com, kelly.kimberly@enron.com, |Y|
Batchnorm
Embed
karen.denne@enron.com, christie.patrick@enron.com,
RELU
Email
rosalee.fleming@enron.com (a) Dropout,σ
Date: Mon, 12 Nov 2001 16:51:33 -0800 (PST)
Ken,
500
I wanted to let you know that I spoke with W.O. King at the Baker Institute earlier
today. Everything is ready for tomorrow's activities.
representation
Email
There will be a great deal of media coverage for the event. Many media outlets will
be accessing the live video feed of Chairman Greenspan's speech. However, the
500 300
media will not be able to interview any participants directly. Chairman Greenspan
Batchnorm
will only answer written questions submitted by the audience (including media), and
(b) 1
Dropout
RELU
RELU
those questions will be vetted twice. I will be receiving the media advisory that has 500
been distributed and the most current media attendee list from the Baker Institute Dropout,σ
tomorrow. I will provide that to you as early as possible.
Target Entity
Candidate
You probably know that, because of his position, Chairman Greenspan can not
accept 2:
Figure theExample
prize itself, but only the
email "honor"
with of being, named
sender an Enron, Prize
recipients keyrecipient.
phrases ,
For that reason, the Enron Prize will not be present on stage during the ceremony.
and replied-to annotations. Each task is constructed by obscuring
Karen Denne, Christie Patrick and I will be attending. Please let me know if you
a relevant entity, then reconstructing it given the remaining context.
have any questions or need additional information prior to the event. |V|
300 σ |Y1|
See you there, 100
Batchnorm
Terrie
computation requirements. So we handle emails differently, com-
Embed
Email
RELU
puting representations on demand through compositions of other (c) Dropout,σ |Y2|
entity representations. In this paper, we explored four different
email composition models: Centroid, Pointwise Max, Pseudoin- σ |Y3|
verse, and the combination of Centroid and Pseudoinverse.
3.2.1 Centroid. One simple email representation is the average Figure 3: Task-specific neural network architectures: (a) multiclass
of the representations of all key phrases and contacts in an email. model for predicting which entity is present in a given email; (b)
3.2.2 Pointwise Max. Another commonly used pooling opera- binary matching model for predicting whether a given entity is
tion is max – we retain the largest value along each dimension. This present in a given email; (c) multi-task multiclass model jointly
trained on all evaluation tasks.
approach increases the sensitivity to strongly-weighted features in
the underlying key phrase and contact representations.
email. We use the cosine similarity between the email represen-
3.2.3 Pseudoinverse. The H matrix from Equation 3 can serve tation and the twenty candidate target entity representations to
as a map from the low-rank concept space into the word/entity predict the true target. Reply prediction is treated as a binary clas-
space. Although H is not a square matrix and hence not invertible, sification problem, using email representations as input features.
the Moore-Penrose pseudoinverse of H , namely H + , can act as a Entity representation methods that yield more accurate predictions
map from email content into the entity representation space. We are considered superior.
multiply the TF-IDF vector associated with a given email by H + to These tasks readily suggest real life applications. Recipient rec-
project into the entity representation space. Unlike the previous ommendation is already a standard feature in many email clients.
two models, this has the benefit of including information from Similarly, an email client may predict whether an email from an
non-key phrase unigrams from the email. unfamiliar address comes from a known sender and prompt the
3.2.4 Centroid + Pseudoinverse. Centroid and pseudoinverse user to add the new address to that sender’s contact information.
representations are summed to combine the benefits of each. Predicting latent associations between emails and key phrases en-
ables automatic topic tagging and foldering. Finally, an email client
4 EVALUATION METHODOLOGY may use reply prediction to identify important emails to which an
inbox owner has not yet responded and remind the user to reply.
4.1 Evaluation Tasks
We evaluate entity representations according to their performance 4.1.1 Task-Specific Model Architectures. We aim to construct
on four email mining tasks: sender prediction, recipient prediction, task-agnostic user-conditioned representations: they should be use-
related key phrase prediction, and reply prediction. The first three ful across a variety of tasks without having to be tuned to each
tasks are content prediction tasks, whereas in reply prediction we one separately. While this makes the representations reusable and
use the email content to predict a user action. reduces computational expense, separate models trained on each
Content prediction tasks are formulated as association tasks. We specific task often perform better. To evaluate this tradeoff, we com-
remove a target entity from an email and randomly select nineteen pare the unsupervised similarity-based method described above
distractor entities from the user’s inbox not already present in the to supervised task-specific baseline models trained on each of the
Privacy-Aware Personalized Entity Representations for Improved User Understanding PrivateNLP 2020, Feb 7, 2020, Houston, Texas
association tasks. We also evaluate how well the user-conditioned Avocado (55 users) Enterprise (53 users)
representations perform as feature inputs to task-specific models, Max Min Average Max Min Average
since their utility as feature representations is a key consideration.
To train a task-specific model for sender, recipient, or key phrase Emails/User 19,000 3,561 7,887 17,490 2,872 8,451
Phrases/User 9,632 3,324 5,308 8,137 3,433 6,772
prediction, we reformulate these association tasks as classification
Contacts/User 376 95 210 2,375 357 1,431
problems. In each case, we train the classifier to predict the target Reply Rate 0.34 0.01 0.14 0.60 0.01 0.19
entity using its email representation. As above, we remove a target
Table 1: Email statistics for Avocado and enterprise users.
entity from an email and select nineteen distractors. Instead of
cosine similarity, we use the trained classifier to score the twenty
candidate entities and predict the one with the highest score.
We experimented with a variety of modeling techniques for both report the average recall, an efficient measure for skewed distri-
task-specific baseline models and task-specific models trained on butions. To obtain the average recall, we calculate the recall for
entity representations. The best results consistently came from each possible target: the percentage of times it was successfully
simple two-layer feed forward neural classifiers using ReLU activa- predicted. We then report the average recall over all targets with-
tions, a sigmoid output layer, batch normalization, drop out [34], out weighing the frequency of the target. Together, accuracy and
and trained using cross-entropy loss and Adam [14]. However, each average recall provide a reliable measure of the association. If one
scenario achieved best results using slightly different task formula- method boosts accuracy by only learning about frequent targets,
tions and architectures. the average recall will be impacted negatively. Similarly a reduced
The baseline models were formulated as multiclass classifiers, recall of the frequent targets will impact the accuracy.
as depicted in Figure 3a. Emails are represented as binary vec- For reply prediction, we report the area under the precision-recall
tors with each element representing the presence or absence of a curve (PR-AUC), which is useful even when classes are imbalanced.
unigram or contact. These vectors index into a 300 dimensional
embedding layer initialized with pre-trained GloVe vectors [31]; out- 4.3 Evaluation Corpora
of-vocabulary items received random initializers. The embedding We evaluate our techniques on two separate repositories of emails,
layer was also trained, allowing the model to learn representations Avocado emails and live user emails from a large enterprise. The
for out-of-vocabulary terms. We experimented with two variants: properties for each corpus are listed in Table 1. For the first reposi-
one in which contacts were included as features (“Pre-trained + tory, we use mailboxes from the Avocado Research Email Collec-
Contacts”) and one in which they were not (“Pre-trained”). tion1 . For the second dataset, we use live user email data from a
For models trained on entity representations, shown in Figure 3b, real-world enterprise with thousands of users (called enterprise
we found the best results by treating the candidate target entity users from here on for brevity). These emails are encrypted and off-
representations and the email representations as separate inputs. limits to human inspection. We randomly select a set of users who
These 500 dimensional representations are passed through two are related to each other by sampling from the same department.
dense layers of width 500 and 300 respectively and a sigmoid out- This increases the possibility of overlap between users and allows
put layer, which returns a score representing the likelihood that some shared context. This property will be helpful when we want
the input entity is indeed present in the input email. In Tables 4 to compare a global model versus user-conditioned representations.
and 5, “TF-IDF + NMF Centroid” and “TF-IDF + NMF Centroid + For both datasets, we filter out users with fewer than 3,500 or
Pseudoinverse” model variants both share this architecture. greater than 20,000 emails. Users with more than 20,000 emails were
We also considered a multitask model jointly trained on all four outliers and, in the enterprise dataset, were likely to have many
evaluation tasks. This model, shown in Figure 3c, is identical in machine generated emails, which can make the evaluation tasks
its architecture and training to the task-specific baseline model in easier. We set the minimum number of emails to 3,500 somewhat
Figure 3a except that instead of one output layer it has |N | output arbitrarily because in our enterprise scenario it is almost always
layers corresponding to the |N | tasks. Relative loss weights were possible to obtain this many for a given user by extending the date
used to balance the training impact from each task since the tasks range. We plan to investigate the performance of user-conditioned
had varying numbers of training examples. representations produced from smaller inboxes in future work.
5 EXPERIMENTS
4.2 Evaluation Metrics We show that user-conditioned entity representations outperform
We measure our performance on the association tasks through strong global model baselines. NMF applied to our version of TF-
accuracy (percentage of successful predictions) and average recall. IDF matrices proves most effective among the methods surveyed for
A successful prediction is one where the target entity is scored representing key phrases and contacts. The combination of centroid
highest among all candidates. Since there is one target and nineteen and pseudoinverse methods detailed in Section 3.2.4 works best for
distractors, random guessing achieves an accuracy of 0.05. composing email representations. While on some tasks supervised
Accuracy can allow a small number of frequently occurring task-specific baseline models achieved higher accuracy than entity
entities to have a disproportionate effect. For instance, in sender representation similarity-based methods, the latter were competi-
prediction the majority of emails may be from a small set of senders: tive and had significantly better recall. Task-specific models trained
performance on these senders will skew the results. Thus, we also 1 https://catalog.ldc.upenn.edu/LDC2015T03
PrivateNLP 2020, Feb 7, 2020, Houston, Texas Melnick, Elmessilhy, Polychronopoulos, Lopez, Tu, Khan, Wang, and Quirk
Sender Recipient Rel. Phrase representations provides the best results for accuracy and almost
Method
matches pseudoinverse for recall.
Acc Rec Acc Rec Acc Rec
TF-IDF 0.59 0.28 0.59 0.31 0.60 0.41 5.2 Task-Specific Models
LDA 0.53 0.37 0.51 0.41 0.49 0.42
Task-Agnostic vs. Task-Specific. Unsupervised, task-agnostic ap-
LSA 0.59 0.29 0.59 0.32 0.60 0.42
proaches are versatile and reusable, but they may underperform
NMF unreg.(λ = 0) 0.61 0.37 0.59 0.40 0.60 0.46
relative to supervised models tuned to specific tasks. As described
NMF (λ = 0.0001) 0.62 0.40 0.62 0.44 0.66 0.53
in Section 4.1.1, we explore this tradeoff by comparing the perfor-
Table 2: Evaluation task performance of key phrase and contact mance of entity representation similarity-based methods against
representation methods. In every case, the tasks use the Centroid task-specific baseline models trained on the evaluation tasks. For
method for composing email representations. Avocado data set.
Avocado, we see that while the accuracy is indeed better on task-
specific Pre-trained and Pre-trained + Contacts compared to the best
representation methods (TF-IDF + NMF Centroid and TF-IDF NMF
Sender Recipient Rel. Phrase Centroid + Pseudoinverse), 3 as shown in Table 4. However, the
Method
Acc Rec Acc Rec Acc Rec TF-IDF + NMF Centroid + Pseudoinverse representations achieved
significantly better recall for all three content prediction tasks and
Centroid 0.62 0.40 0.62 0.44 0.66 0.53
better accuracy in key phrase prediction, again indicating their
Pointwise max 0.59 0.30 0.59 0.34 0.61 0.42
ability to avoid over-optimizing for frequently occurring entities.
Pseudoinverse 0.49 0.56 0.47 0.56 0.55 0.58
This model produces even better results on the enterprise data
Centroid+Pseudoinv 0.64 0.53 0.62 0.54 0.66 0.53
set, where its accuracy is competitive with both of the Pre-trained
Table 3: Evaluation task performance of email representation meth- models and its improvement in recall is even more dramatic. The
ods. In every case, the tasks use regularized NMF to produce key higher number of contacts in the enterprise set enables better joint
phrase and contact representations. Avocado data set.
modeling with the content, allowing the entity representations to
perform better in this setting. We can see that unsupervised entity
representations are competitive with supervised baselines.
on entity representations also outperformed task-specific models Entity Representations as Input Features. As our results suggest,
trained on baseline features, demonstrating the entity representa- user-conditioned entity representations are useful as input features
tions’ value as feature inputs. Our results here also show that entity to supervised models. To assess their value as feature representa-
representations are competitive with multitask learning despite the tions, we compare task-specific models trained on entity represen-
fact that they are trained without knowledge of the downstream tations with task-specific baselines, as described in Section 4.1.1.
tasks. We discuss these results in the following subsections. On Avocado, the entity representation-based task-specific models,
TF-IDF + NMF Centroid and TF-IDF NMF Centroid + Pseudoin-
5.1 User-Conditioned Representations verse, outperform (or in a few cases match) the baselines on every
Slow Changing Entities: Key Phrases and Contacts. We compare task and metric. We see similar results on enterprise data, except a
unsupervised methods for producing key phrase and contact repre- marginally lower reply prediction PR-AUC with entity-based task-
sentations in Table 2. For LDA, LSA, and NMF, we perform hyper- specific models. Comparing the Avocado and enterprise results,
parameter tuning on a single enterprise user and report results for we can see that the performance on all tasks is much better on
all techniques with their best settings. Since the evaluation tasks re- enterprise users. Our hypothesis is that the larger contact vocabu-
quire representations for email as well as key phrases and contacts, lary in enterprise (1,431 contacts per user on average) compared
we use the Centroid email representation in each case to ensure a to Avocado (average 210 contacts per user) makes sender and re-
fair comparison. Predictions are based on cosine similarity.2 NMF cipient tasks easier: the distractors are sampled from a larger pool
with regularization outperformed all other methods. Regularization of contacts, and therefore less likely to be related and easier to
leads to more effective representations for NMF; comparing unreg- screen out. In the case of reply prediction, we believe the higher
ularized NMF to LSA suggests that non-negativity is also a helpful PR-AUC stems from enterprise users that receive a higher volume
bias. Some of the most substantial gains are in recall, especially of machine-generated emails, which have more predictable reply
when compared to sparse TF-IDF baselines. behavior.
Composition for Fast Changing Entities: Email. Different compo-
sitional operations for representing email are explored in Table 3. 5.3 User-Conditioned vs. Global Models
Because NMF performed best across all tasks, we restrict our at-
Each set of user-conditioned representations is trained on much
tention to these representations. The centroid method outperforms
fewer data than most representation learning techniques, but per-
others on accuracy, though the pseudoinverse approach is the best
sonalization is a powerful source of context. While our primary
for recall, presumably because it can incorporate information from
reason for focusing on user-conditioned entity representations is
unigrams in the represented email and not just the key phrases
to avoid privacy leaks, we want to know how they compare against
and contacts. A linear combination of centroid and pseudoinverse
2 Reply prediction is difficult to evaluate in an unsupervised setting; hence, it is not 3 Our results for sender and recipient prediction through an unsupervised task-agnostic
reported here. representation are in the same range as those reported by Graus et al., [17] (0.66).
Privacy-Aware Personalized Entity Representations for Improved User Understanding PrivateNLP 2020, Feb 7, 2020, Houston, Texas
Sender Recipient Related Phrase Reply
Data Method
Accuracy Recall Accuracy Recall Accuracy Recall PR-AUC
Unsupervised Similarity-Based Methods
TF-IDF + NMF Centroid 0.62 0.40 0.62 0.44 0.66 0.53 N/A
TF-IDF + NMF Centroid + Pseudoinverse 0.64 0.53 0.62 0.54 0.67 0.60 N/A
Supervised Task-Specific Models
Avocado users
Pre-trained 0.72 0.38 0.67 0.31 0.59 0.36 0.21
Pre-trained + Contacts 0.74 0.42 0.71 0.35 0.60 0.37 0.24
TF-IDF + NMF Centroid 0.74 0.48 0.72 0.47 0.64 0.49 0.28
TF-IDF + NMF Centroid + Pseudoinverse 0.74 0.49 0.73 0.47 0.67 0.52 0.28
Supervised Multi-Task Models
Pre-trained 0.73 0.47 0.69 0.42 0.59 0.35 0.28
Pre-trained + Contacts 0.78 0.51 0.75 0.46 0.59 0.35 0.30
Unsupervised Similarity-Based Methods
TF-IDF + NMF Centroid 0.81 0.73 0.86 0.79 0.69 0.60 N/A
Enterprise users
TF-IDF + NMF Centroid + Pseudoinverse 0.81 0.77 0.86 0.81 0.70 0.65 N/A
Supervised Task-Specific Models
Pre-trained + Contacts 0.83 0.54 0.87 0.50 0.70 0.44 0.71
TF-IDF + NMF Centroid 0.87 0.68 0.91 0.71 0.72 0.56 0.69
TF-IDF + NMF Centroid + Pseudoinverse 0.87 0.70 0.91 0.72 0.74 0.59 0.65
Supervised Multi-Task Models
Pre-trained + Contacts 0.85 0.58 0.88 0.54 0.70 0.43 0.72
Table 4: Task-specific models trained using representations as features, for both enterprise and Avocado users.
non-privacy-aware “global” representations trained on data from ev-
ery user in an organization. In Table 5 we see that user-conditioned
representations are significantly better on all tasks across all met-
rics compared to the global versions of those representations. This
indicates that, for these models, the local context of a user is more
important than training on a larger data set. We see a similar trend
with the Pre-trained + Contacts and Global Pre-trained + Contacts
models, though the global variant outperforms the user-conditioned
one in sender prediction on Avocado. On reply prediction, global Figure 4: Sender prediction accuracy vs. number of training emails
models trained using representations perform similarly to Yang et for TF-IDF + NMF Centroid on Avocado.
al. [40] without any task-specific feature engineering.
Thus the unsupervised methods presented here are competitive
with multitask learning on recall despite the fact that they are
5.4 Unsupervised vs. Multi-Task Approaches trained without knowledge of the downstream tasks, and the task-
Our primary focus has been unsupervised entity representation specific entity-based models are competitive with the multi-task
computation. An alternative approach is to induce representations method on accuracy and better on recall.
in a multitask learning setting [9]. Multitask models often achieve
better performance than separate models trained on the same tasks 5.5 The Effect of Data Size and Dimension
and, indeed, as seen in Table 4, the multitask model described in To explore the impact of data density, Figure 4 plots sender pre-
Section 4.1.1 outperforms task specific models trained on the same diction accuracy using TF-IDF + NMF Centroid representations
Pre-trained + Contacts feature representation. against the number of emails in a user’s mailbox. Accuracy does
On Avocado, the best multitask model achieves significantly bet- not vary substantially, though average recall improves: additional
ter accuracy in sender and recipient prediction than Pre-trained and data benefits representing entities on average. Similar trends for
Pre-trained + Contacts methods; entity-based task-specific methods other tasks and other models were observed.
are still competitive on recall. We observe the same trend with enter- We plot the effect of varying dimension sizes for all tasks using
prise, where multi-task models outperform task-specific Pre-trained the TF-IDF + NMF Centroid approach in Figure 5 for Avocado users.
+ Contacts, though entity-based task-specific models outperform Representations of dimension 400 and 500 consistently achieve best
multi-task on all tasks and metrics except reply prediction PR-AUC. results for both accuracy and recall.
PrivateNLP 2020, Feb 7, 2020, Houston, Texas Melnick, Elmessilhy, Polychronopoulos, Lopez, Tu, Khan, Wang, and Quirk
Sender Recipient Related Phrase Reply
Data Method
Accuracy Recall Accuracy Recall Accuracy Recall PR-AUC
Unsupervised Similarity-Based Methods
TF-IDF + NMF Centroid 0.62 0.40 0.62 0.44 0.66 0.53 N/A
TF-IDF + NMF Centroid + Pseudoinverse 0.64 0.53 0.62 0.54 0.67 0.60 N/A
Global TF-IDF + NMF 0.50 0.29 0.41 0.27 0.45 0.30 N/A
Avocado users
Global TF-IDF + NMF Centroid + Pseudoinverse 0.55 0.40 0.40 0.34 0.43 0.37 N/A
Supervised Task-Specific Models
Pre-trained + Contacts 0.74 0.42 0.71 0.35 0.60 0.37 0.24
TF-IDF + NMF Centroid 0.74 0.48 0.72 0.47 0.64 0.49 0.28
TF-IDF + NMF Centroid + Pseudoinverse 0.74 0.49 0.73 0.47 0.67 0.52 0.28
Global Pre-trained + Contacts 0.77 0.63 0.65 0.48 0.58 0.34 0.21
Global TF-IDF + NMF Centroid 0.70 0.50 0.58 0.36 0.52 0.29 0.25
Global TF-IDF + NMF Centroid + Pseudoinverse 0.71 0.52 0.57 0.37 0.53 0.30 0.19
Unsupervised Similarity-Based Methods
TF-IDF + NMF Centroid 0.81 0.73 0.86 0.79 0.69 0.60 N/A
TF-IDF + NMF Centroid + Pseudoinverse 0.81 0.77 0.86 0.81 0.70 0.65 N/A
Global TF-IDF + NMF 0.50 0.49 0.45 0.30 0.45 0.41 N/A
Enterprise users
Global TF-IDF + NMF Centroid + Pseudoinverse 0.48 0.51 0.43 0.34 0.45 0.43 N/A
Supervised Task-Specific Models
Pre-trained + Contacts 0.83 0.54 0.87 0.50 0.70 0.44 0.71
TF-IDF + NMF Centroid 0.87 0.68 0.91 0.71 0.72 0.56 0.69
TF-IDF + NMF Centroid + Pseudoinverse 0.87 0.70 0.91 0.72 0.74 0.59 0.65
Global Pretrained + Contacts 0.80 0.61 0.77 0.44 0.63 0.46 0.65
Global TF-IDF + NMF Centroid 0.61 0.49 0.47 0.25 0.49 0.34 0.67
Global TF-IDF + NMF Centroid + Pseudoinverse 0.60 0.51 0.48 0.30 0.50 0.36 0.56
Table 5: Individual vs. global models on Avocado and enterprise users.
approaches are privacy preserving and show substantial benefits
over global models, despite their lower data density. These promis-
ing results suggest a range of future directions to explore. One
clear next step is to extend our approach to include documents,
meetings, and other enterprise entities. Beyond that, embedding
relationships between entities could help in predicting more com-
plex connections between them. Next, our explorations in multitask
modeling suggest that generalization across tasks also has value.
Evaluating the impact of multitask representations on new tasks
Figure 5: Effect of dimensionality on entity representations.
through leave-one-out experiments may help quantify this.
5.6 Practical Implications
Our current implementation has several optimizations intended for
REFERENCES
[1] Q. Ai, V. Azizi, X. Chen, and Y. Zhang. Learning heterogeneous knowledge base
a production environment. We maintain updates to the TF-IDF ma- embeddings for explainable recommendation. Algorithms, 11(9):137, 2018.
trix in a streaming manner upon receipt of each email. A periodic [2] Q. Ai, Y. Zhang, K. Bi, X. Chen, and W. B. Croft. Learning a hierarchical embedding
task, run every few days to every week, computes a fresh NMF rep- model for personalized product search. In Proceedings of the 40th International
ACM SIGIR Conference on Research and Development in Information Retrieval,
resentation, using approximately one minute of computation time pages 645–654, 2017.
per user with an optimized implementation based on Sparse BLAS [3] N. O. Amer, P. Mulhem, and M. Géry. Toward word embedding for personalized
information retrieval. CoRR, abs/1606.06991, 2016.
operations in Intel MKL. [20] The process is running constantly for [4] R. Balasubramanyan, V. R. Carvalho, and W. Cohen. Cutonce-recipient rec-
thousands of users, scaling up to hundreds of thousands of users. ommendation and leak detection in action. In AAAI, Workshop on Enhanced
Messaging, 2008.
[5] Y. Bao, H. Fang, and J. Zhang. TopicMF: Simultaneously exploiting ratings and
6 CONCLUSIONS reviews for recommendation. In Twenty-Eighth AAAI Conference on Artificial
We have demonstrated approaches for learning task-agnostic user- Intelligence, 2014.
[6] P. N. Bennett and J. G. Carbonell. Combining probability-based rankers for action-
conditioned embeddings that outperform strong baselines and item detection. In Human Language Technologies: The Conference of the North
demonstrate value in a range of downstream tasks. User-conditioned American Chapter of the Association for Computational Linguistics; Proceedings of
Privacy-Aware Personalized Entity Representations for Improved User Understanding PrivateNLP 2020, Feb 7, 2020, Houston, Texas
the Main Conference, pages 324–331, 2007. [36] K. Toutanova, D. Chen, P. Pantel, H. Poon, P. Choudhury, and M. Gamon. Rep-
[7] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. J. Mach. Learn. resenting text for joint embedding of text and knowledge bases. In EMNLP,
Res., 3:993–1022, Mar. 2003. 2015.
[8] A. Bratko, B. Filipič, G. V. Cormack, T. R. Lynam, and B. Zupan. Spam filtering [37] C. Van Gysel, B. Mitra, M. Venanzi, R. Rosemarin, G. Kukla, P. Grudzien, and
using statistical data compression models. JMLR, 7:2673–2698, Dec. 2006. N. Cancedda. Reply with: Proactive recommendation of email attachments.
[9] R. Caruana. Multitask learning. Mach. Learn., 28(1):41–75, July 1997. In Proceedings of the 2017 ACM on Conference on Information and Knowledge
[10] M. X. Chen, B. N. Lee, G. Bansal, Y. Cao, S. Zhang, J. Lu, J. Tsay, Y. Wang, A. M. Management, CIKM, pages 327–336, New York, NY, USA, 2017.
Dai, Z. Chen, T. Sohn, and Y. Wu. Gmail smart compose: Real-time assisted [38] J. B. P. Vuurens, M. Larson, and A. P. de Vries. Exploring deep space: Learning
writing. In Proceedings of the 25th ACM SIGKDD International Conference on personalized ranking in a semantic space. In Proceedings of the 1st Workshop on
Knowledge Discovery & Data Mining, KDD ’19, pages 2287–2295, New York, Deep Learning for Recommender Systems, DLRS 2016, 2016.
NY, USA, 2019. ACM. [39] L. Y. Wu, A. Fisch, S. Chopra, K. Adams, A. Bordes, and J. Weston. Starspace:
[11] S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman. In- Embed all the things! In AAAI, 2018.
dexing by latent semantic analysis. Journal of the American Society for Information [40] L. Yang, S. T. Dumais, P. N. Bennett, and A. H. Awadallah. Characterizing and
Science, 41(6):391–407, 1990. predicting enterprise email reply behavior. In Proceedings of the 40th International
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of ACM SIGIR Conference on Research and Development in Information Retrieval,
deep bidirectional transformers for language understanding. arXiv preprint pages 235–244, 2017.
arXiv:1810.04805, 2018. [41] S. Yoo, Y. Yang, F. Lin, and I.-C. Moon. Mining social networks for personal-
[13] F. Diaz, B. Mitra, and N. Craswell. Query expansion with locally-trained word ized email prioritization. In Proceedings of the 15th ACM SIGKDD international
embeddings. CoRR, abs/1605.07891, 2016. conference on Knowledge discovery and data mining, pages 967–976, 2009.
[14] J. B. Diederik Kingma. Adam: A method for stochastic optimization. In
arXiv:1412.6980, 2014.
[15] M. Dredze, T. Brooks, J. Carroll, J. Magarick, J. Blitzer, and F. Pereira. Intelligent
email: Reply and attachment prediction. In Proceedings of the 13th International
ACM Conference on Intelligent user interfaces, pages 321–324, 2008.
[16] J. Goodman, G. V. Cormack, and D. Heckerman. Spam and the ongoing battle for
the inbox. Communications of the ACM, 50(2):24–33, 2007.
[17] D. Graus, D. Van Dijk, M. Tsagkias, W. Weerkamp, and M. De Rijke. Recipient
recommendation in enterprises using communication graphs and email content.
In Proceedings of the 37th international ACM SIGIR conference on Research &
development in information retrieval, pages 1079–1082, 2014.
[18] K. S. Hasan and V. Ng. Automatic keyphrase extraction: A survey of the state
of the art. In Proceedings of the 52nd Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 1262–1273, Baltimore,
Maryland, June 2014. Association for Computational Linguistics.
[19] G. Hu. Personalized neural embeddings for collaborative filtering with text. arXiv
preprint arXiv:1903.07860, 2019.
[20] Intel. Intel Math Kernel Library. Reference Manual. Intel Corporation, Santa Clara,
USA, 2009. ISBN 630813-054US.
[21] J. Kim, Y. He, and H. Park. Algorithms for nonnegative matrix and tensor
factorizations: a unified view based on block coordinate descent framework.
Journal of Global Optimization, 58(2):285–319, Feb 2014.
[22] B. Klimt and Y. Yang. The enron corpus: A new dataset for email classification
research. In European Conference on Machine Learning. Springer, 2004.
[23] S. Kottur, X. Wang, and V. Carvalho. Exploring personalized neural conversational
models. In IJCAI, pages 3728–3734, 2017.
[24] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep
convolutional neural networks. In Advances in neural information processing
systems, pages 1097–1105, 2012.
[25] D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix
factorization. Nature, 401:788–791, 1999.
[26] M. Levit, A. Stolcke, R. Subba, S. Parthasarathy, S. Chang, S. Xie, T. Anastasakos,
and B. Dumoulin. Personalization of word-phrase-entity language models. In
Proc. Interspeech, pages 448–452. ISCA - International Speech Communication
Association, September 2015.
[27] S. Liang, X. Zhang, Z. Ren, and E. Kanoulas. Dynamic embeddings for user pro-
filing in twitter. In Proceedings of the 24th ACM SIGKDD International Conference
on Knowledge Discovery & Data Mining, pages 1764–1773, 2018.
[28] T. Mikolov, I. Sutskever, K. Chen, G. Corrado, and J. Dean. Distributed represen-
tations of words and phrases and their compositionality. In Advances in Neural
Information Processing Systems.
[29] W. Nawaz, Y. Han, K.-U. Khan, and Y.-K. Lee. Personalized email community
detection using collaborative similarity measure. arXiv preprint:1306.1300, 2013.
[30] T. Nguyen and A. Takasu. Npe: neural personalized embedding for collaborative
filtering. arXiv preprint arXiv:1805.06563, 2018.
[31] J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word repre-
sentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, Oct. 2014. Associa-
tion for Computational Linguistics.
[32] A. Rattinger, J.-M. L. Goff, and C. Gütl. Local word embeddings for query expan-
sion based on co-authorship and citations. In BIR@ECIR, 2018.
[33] M. Rudolph, F. Ruiz, S. Mandt, and D. Blei. Exponential family embeddings. In
Advances in Neural Information Processing Systems, pages 478–486, 2016.
[34] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.
Dropout: A simple way to prevent neural networks from overfitting. Journal of
Machine Learning Research, 15:1929–1958, 2014.
[35] G. Tang, J. Pei, and W.-S. Luk. Email mining: tasks, common techniques, and
tools. Knowledge and Information Systems, 41(1):1–31, 2014.