Representation learning has transformed the field of machine learning. Advances like ImageNet, word2vec, and BERT demonstrate the power of pre-trained representations to accelerate model training. The efectiveness of these techniques derives from their ability to represent words, sentences, and images in context. Other entity types, such as people and topics, are crucial sources of context in enterprise use-cases, including organization, recommendation, and discovery of vast streams of information. But learning representations for these entities from private data aggregated across user shards carries the risk of privacy breaches. Personalizing representations by conditioning them on a single user's content eliminates privacy risks while providing a rich source of context that can change the interpretation of words, people, documents, groups, and other entities commonly encountered in workplace data. In this paper, we explore methods that embed user-conditioned representations of people, key phrases, and emails into a shared vector space based on an individual user's emails. We evaluate these representations on a suite of representative communication inference tasks using both a public email repository and live user data from an enterprise. We demonstrate that our privacy-preserving lightweight unsupervised representations rival supervised approaches. When used to augment supervised approaches, these representations are competitive with deep-learned multi-task models based on pre-trained representations.
Pre-trained embeddings are a crucial technique in machine
learning applications, especially when task-specific training data is
scarce. For instance, groundbreaking work in image captioning
was enabled by reusing the penultimate layer of an object
recognition system to summarize the content of an image[
Our research is situated in the area of User Understanding: organizing the information, documents, and communications that are available to each user within an organization. Users now commonly retain huge mailboxes of written communication; members of larger organizations also have access to large repositories of access-controlled documents that are not publicly available. Our goals are to help each user find, classify, and act upon these growing information stores, then to acquire and organize information, including facts and relationships among these entities. A crucial enabling step is to build reusable representations of this information.
Most representation learning uses large, publicly-available document stores to build generic embeddings. We believe there is also great value in user-conditioned representations: representations of phrases and contacts for each user learned on the information uniquely available to that user. First, building user-conditioned representations provides a huge amount of context. Often when there are ambiguous or overloaded concepts, the key people surrounding their usage can disambiguate. Furthermore, a given user may extend the meanings of a given concept as they document and communicate new ideas. Perhaps most importantly, training a model based on only the communications and documents available to a given user provides a clear and intuitive notion of privacy. Whenever we train on data beyond any user’s normal visibility, there is some potential for capturing and surfacing information outside their view. Diferential Privacy helps limit the exposure of any individual user, but preventing leakage across groups is more dificult. For instance, certain privileged information may be discussed heavily by many members of an administrative board, yet this information should not be shared broadly across the whole organization. When training a user’s model on only data that that user can see, the possibility for leaking information is removed. From the perspectives of both leveraging a crucial signal as well as maintaining user privacy and trust, user-conditioned representations hold great promise.
User-conditioned learning comes at a cost. Data density
decreases dramatically. State-of-the-art deep learned representations
typically train on billions of tokens [
As computation and storage become cheaper, the overhead of maintaining user-conditioned models is tractable only if the models are light-weight. Furthermore, we focus on task-agnostic representations that benefit a range of scenarios, amortizing the cost of computation. Finally, using models trained only on one user’s data benefits privacy, which is an increasing concern for organizations and individuals. 1.1
We present the first eforts in building per-user representations: content-based representations that embed disparate entities, including contacts and key phrases, into a common vector space. These entity representations are diferent for each user: the same key phrase and contact may have very diferent representations across two users depending on their context. We focus on slow-changing entities like contacts and key phrases to minimize the impact of delayed retraining: although one’s impression of their collaborators may shift over years, months, or perhaps weeks, a representation that is a few days old is still useful. To embed rapidly arriving and changing items such as documents and emails, we present approaches that assemble representations of rapidly changing entities from their content, including related contacts and key phrases.
We evaluate these representations on a range of downstream
tasks, including action-prediction and content-prediction. Simple,
unsupervised approaches, especially non-negative matrix
factorization (NMF) [
We compare user-conditioned representations to representations learned at the organization-level, where data from multiple users are combined together into a larger undiferentiated store. Userconditioned representations mostly outperform the organizationlevel approaches, despite decreased data density, presumably because the additional context provides helpful signals to models. Furthermore, user-conditioned representations sidestep issues related to privacy preservation by not mixing data across user mailboxes. 2
Email mining [
Entity representations have been used extensively to provide
personalized recommendations. Most such models build global
representations of users and items in the same latent space and then
determine the similarity between the user and item through cosine
similarity. The user embeddings can be built in a collaborative
filtering setting by leveraging a user’s past actions such as clicks [
Personalized language models have shown benefits in speech
recognition [
Starspace [
Our evaluation tasks bear some similarity to the evaluation of
knowledge base completion through embeddings of text and
entities [
In query expansion, locally trained embeddings can outperform
global or pre-trained embeddings [
We developed representations for three entity types: key phrases,
contacts, and emails. Key phrases (typically noun phrases) [
Since the median inbox size will be small (in both our data sets it
is around seven thousand emails) there is not enough data per user
to train useful deep learned representations. Indeed, our early
attempts to train user-conditioned word2vec [
de,f = We compute unsupervised entity representations for contacts and key phrases by associating them with concatenated documents. These concatenated documents are assembled from the user’s original documents; the emails in our experiments have From, To, CC, Body, and Subject fields. Given a particular entity e, its concatenated document de is the concatenation of every email m in a user’s inbox such that e appears in any of those fields. This concatenation is done on each field f independently: every new concatenated document de will have a corresponding field de,f for each field f in the original document. We use É to denote concatenation. Ê mf , where Me = {m : ∃f such that e ∈ mf } (1) m ∈Me Stop words on the Scikit-learn stopword list are removed, as are terms that appear in more than 30% or less than 0.25% of training emails. We then generate a sparse numerical entity-by-term matrix, using TF-IDF for most methods or just TF matrix in the case of LDA. Initially one matrix is computed for each field f using the relevant portion of the concatenated document collection Df = {de,f }e . Each matrix is scaled according to a weighting factor wf to balance its contributions, and finally these matrices are concatenated to form a single matrix T .
T = Ê pwf · term-matrix(Df ), where Õ wf = 1 (2) f f The weights of the diferent email fields are treated as hyperparameters and tuned empirically to perform well on the evaluation tasks. We found that weights of 0.4, 0.3, 0.2, 0.05, and 0.05 for the Body, Subject, From, To, and CC fields worked well. The rows of T are sparse representations of entities – a simple and safe baseline. We explored LDA, LSA, and NMF as a means of encouraging softer matching through dimensionality reduction.
3.1.1 TF-IDF. Our baseline representation technique is sparse unigram TF-IDF vectors produced from the concatenated documents.
3.1.2 Latent Dirichlet Allocation (LDA). Latent topic models
using LDA [
3.1.3 Latent Semantic Analysis (LSA). A classic method for
reducing sparsity, LSA [
3.1.4 Non-negative Matrix Factorization (NMF). The SVD reconstruction has a few problems: the values of the matrix may be positive or negative, and there is no explicit regularization term. Together, these issues may lead to strange or divergent weights, especially when the data is dificult to model with lower rank. Non-negative matrix factorization (NMF) addresses these issues by E m a li s C o n t a c ts + k e y p h r a s e s C o n t a c ts + k e y p h r a s e s the item … Words + contacts + key phrases mail1 mail2 … mailn due date 4 2 8 the 0.1 0.2 0.3 0 1Original corpus:1 0TF-IDF matrix 0 ⋱
1 item 0.3 0 0
Reduce dimensionality of concatenated document matrix
⋱
H
0.25
0
constraining the low-rank matrices to be positive and adding a
regularization term [
The vocabulary of key phrases and contacts in one’s mailbox is likely to grow slowly, and their meanings and relationships will also evolve gradually. By comparison, many new emails arrive every day, so the “vocabulary” of email entities is constantly increasing. Thus, while it is possible to train representations for email in the same way that we do for key phrases and contacts, updating email representations on an ongoing basis would imply vast storage and
Ken, Replied to: False
today. Everything is ready for tomorrow's activities.
There will be a great deal of media coverage for the event. Many media outlets will be accessing the live video feed of Chairman Greenspan's speech. However, the media will not be able to interview any participants directly. Chairman Greenspan will only answer written questions submitted by the audience (including media), and those questions will be vetted twice. I will be receiving the media advisory that has been distributed and the most current media attendee list from the Baker Institute tomorrow. I will provide that to you as early as possible.
Figauccreept2th:eEpxriazemitspellfe, beumtoanliyl twhei"thhonosre" nofdbeering, narmecedipaineEnntrson,Prkizeeyrecpiphiernats.es ,
and replied-to annotations. Each task is constructed by obscuring a rhKealavereevnaanDnyetnqeunnees,ttCiiohtnyrsi,sottirhenPeeanetdriackddaintidonIawl iilnlfboermatatetinodninpgri.oPrletoasteheleetvmenmetk.anionwiinfygocuontext.
reconstructing it given the re
See you there, comTerpriuetation requirements. So we handle emails diferently, computing representations on demand through compositions of other entity representations. In this paper, we explored four diferent email composition models: Centroid, Pointwise Max, Pseudoinverse, and the combination of Centroid and Pseudoinverse.
3.2.1 Centroid. One simple email representation is the average of the representations of all key phrases and contacts in an email.
3.2.2 Pointwise Max. Another commonly used pooling operation is max – we retain the largest value along each dimension. This approach increases the sensitivity to strongly-weighted features in the underlying key phrase and contact representations.
3.2.3 Pseudoinverse. The H matrix from Equation 3 can serve as a map from the low-rank concept space into the word/entity space. Although H is not a square matrix and hence not invertible, the Moore-Penrose pseudoinverse of H , namely H +, can act as a map from email content into the entity representation space. We multiply the TF-IDF vector associated with a given email by H + to project into the entity representation space. Unlike the previous two models, this has the benefit of including information from non-key phrase unigrams from the email.
3.2.4 Centroid + Pseudoinverse. Centroid and pseudoinverse representations are summed to combine the benefits of each.
We evaluate entity representations according to their performance on four email mining tasks: sender prediction, recipient prediction, related key phrase prediction, and reply prediction. The first three tasks are content prediction tasks, whereas in reply prediction we use the email content to predict a user action.
Content prediction tasks are formulated as association tasks. We remove a target entity from an email and randomly select nineteen distractor entities from the user’s inbox not already present in the (a) (b) (c) |V| li a m E 500 p e r 500 |V| li a m E n o i email. We use the cosine similarity between the email representation and the twenty candidate target entity representations to predict the true target. Reply prediction is treated as a binary classification problem, using email representations as input features. Entity representation methods that yield more accurate predictions are considered superior.
These tasks readily suggest real life applications. Recipient recommendation is already a standard feature in many email clients. Similarly, an email client may predict whether an email from an unfamiliar address comes from a known sender and prompt the user to add the new address to that sender’s contact information. Predicting latent associations between emails and key phrases enables automatic topic tagging and foldering. Finally, an email client may use reply prediction to identify important emails to which an inbox owner has not yet responded and remind the user to reply.
4.1.1 Task-Specific Model Architectures. We aim to construct task-agnostic user-conditioned representations: they should be useful across a variety of tasks without having to be tuned to each one separately. While this makes the representations reusable and reduces computational expense, separate models trained on each specific task often perform better. To evaluate this tradeof, we compare the unsupervised similarity-based method described above to supervised task-specific baseline models trained on each of the association tasks. We also evaluate how well the user-conditioned representations perform as feature inputs to task-specific models, since their utility as feature representations is a key consideration.
To train a task-specific model for sender, recipient, or key phrase prediction, we reformulate these association tasks as classification problems. In each case, we train the classifier to predict the target entity using its email representation. As above, we remove a target entity from an email and select nineteen distractors. Instead of cosine similarity, we use the trained classifier to score the twenty candidate entities and predict the one with the highest score.
We experimented with a variety of modeling techniques for both
task-specific baseline models and task-specific models trained on
entity representations. The best results consistently came from
simple two-layer feed forward neural classifiers using ReLU
activations, a sigmoid output layer, batch normalization, drop out [
The baseline models were formulated as multiclass classifiers,
as depicted in Figure 3a. Emails are represented as binary
vectors with each element representing the presence or absence of a
unigram or contact. These vectors index into a 300 dimensional
embedding layer initialized with pre-trained GloVe vectors [
For models trained on entity representations, shown in Figure 3b, we found the best results by treating the candidate target entity representations and the email representations as separate inputs. These 500 dimensional representations are passed through two dense layers of width 500 and 300 respectively and a sigmoid output layer, which returns a score representing the likelihood that the input entity is indeed present in the input email. In Tables 4 and 5, “TF-IDF + NMF Centroid” and “TF-IDF + NMF Centroid + Pseudoinverse” model variants both share this architecture.
We also considered a multitask model jointly trained on all four evaluation tasks. This model, shown in Figure 3c, is identical in its architecture and training to the task-specific baseline model in Figure 3a except that instead of one output layer it has |N | output layers corresponding to the |N | tasks. Relative loss weights were used to balance the training impact from each task since the tasks had varying numbers of training examples. 4.2
We measure our performance on the association tasks through accuracy (percentage of successful predictions) and average recall. A successful prediction is one where the target entity is scored highest among all candidates. Since there is one target and nineteen distractors, random guessing achieves an accuracy of 0.05.
Accuracy can allow a small number of frequently occurring entities to have a disproportionate efect. For instance, in sender prediction the majority of emails may be from a small set of senders: performance on these senders will skew the results. Thus, we also
Max report the average recall, an eficient measure for skewed distributions. To obtain the average recall, we calculate the recall for each possible target: the percentage of times it was successfully predicted. We then report the average recall over all targets without weighing the frequency of the target. Together, accuracy and average recall provide a reliable measure of the association. If one method boosts accuracy by only learning about frequent targets, the average recall will be impacted negatively. Similarly a reduced recall of the frequent targets will impact the accuracy.
For reply prediction, we report the area under the precision-recall curve (PR-AUC), which is useful even when classes are imbalanced. 4.3
We evaluate our techniques on two separate repositories of emails, Avocado emails and live user emails from a large enterprise. The properties for each corpus are listed in Table 1. For the first repository, we use mailboxes from the Avocado Research Email Collection1. For the second dataset, we use live user email data from a real-world enterprise with thousands of users (called enterprise users from here on for brevity). These emails are encrypted and oflimits to human inspection. We randomly select a set of users who are related to each other by sampling from the same department. This increases the possibility of overlap between users and allows some shared context. This property will be helpful when we want to compare a global model versus user-conditioned representations.
For both datasets, we filter out users with fewer than 3,500 or greater than 20,000 emails. Users with more than 20,000 emails were outliers and, in the enterprise dataset, were likely to have many machine generated emails, which can make the evaluation tasks easier. We set the minimum number of emails to 3,500 somewhat arbitrarily because in our enterprise scenario it is almost always possible to obtain this many for a given user by extending the date range. We plan to investigate the performance of user-conditioned representations produced from smaller inboxes in future work. 5
We show that user-conditioned entity representations outperform strong global model baselines. NMF applied to our version of TFIDF matrices proves most efective among the methods surveyed for representing key phrases and contacts. The combination of centroid and pseudoinverse methods detailed in Section 3.2.4 works best for composing email representations. While on some tasks supervised task-specific baseline models achieved higher accuracy than entity representation similarity-based methods, the latter were competitive and had significantly better recall. Task-specific models trained
on entity representations also outperformed task-specific models trained on baseline features, demonstrating the entity representations’ value as feature inputs. Our results here also show that entity representations are competitive with multitask learning despite the fact that they are trained without knowledge of the downstream tasks. We discuss these results in the following subsections. 5.1
Slow Changing Entities: Key Phrases and Contacts. We compare unsupervised methods for producing key phrase and contact representations in Table 2. For LDA, LSA, and NMF, we perform hyperparameter tuning on a single enterprise user and report results for all techniques with their best settings. Since the evaluation tasks require representations for email as well as key phrases and contacts, we use the Centroid email representation in each case to ensure a fair comparison. Predictions are based on cosine similarity.2 NMF with regularization outperformed all other methods. Regularization leads to more efective representations for NMF; comparing unregularized NMF to LSA suggests that non-negativity is also a helpful bias. Some of the most substantial gains are in recall, especially when compared to sparse TF-IDF baselines.
Composition for Fast Changing Entities: Email. Diferent compositional operations for representing email are explored in Table 3. Because NMF performed best across all tasks, we restrict our attention to these representations. The centroid method outperforms others on accuracy, though the pseudoinverse approach is the best for recall, presumably because it can incorporate information from unigrams in the represented email and not just the key phrases and contacts. A linear combination of centroid and pseudoinverse representations provides the best results for accuracy and almost matches pseudoinverse for recall. 5.2
Task-Agnostic vs. Task-Specific. Unsupervised, task-agnostic approaches are versatile and reusable, but they may underperform relative to supervised models tuned to specific tasks. As described in Section 4.1.1, we explore this tradeof by comparing the performance of entity representation similarity-based methods against task-specific baseline models trained on the evaluation tasks. For Avocado, we see that while the accuracy is indeed better on taskspecific Pre-trained and Pre-trained + Contacts compared to the best representation methods (TF-IDF + NMF Centroid and TF-IDF NMF Centroid + Pseudoinverse), 3 as shown in Table 4. However, the TF-IDF + NMF Centroid + Pseudoinverse representations achieved significantly better recall for all three content prediction tasks and better accuracy in key phrase prediction, again indicating their ability to avoid over-optimizing for frequently occurring entities. This model produces even better results on the enterprise data set, where its accuracy is competitive with both of the Pre-trained models and its improvement in recall is even more dramatic. The higher number of contacts in the enterprise set enables better joint modeling with the content, allowing the entity representations to perform better in this setting. We can see that unsupervised entity representations are competitive with supervised baselines.
Entity Representations as Input Features. As our results suggest, user-conditioned entity representations are useful as input features to supervised models. To assess their value as feature representations, we compare task-specific models trained on entity representations with task-specific baselines, as described in Section 4.1.1. On Avocado, the entity representation-based task-specific models, TF-IDF + NMF Centroid and TF-IDF NMF Centroid + Pseudoinverse, outperform (or in a few cases match) the baselines on every task and metric. We see similar results on enterprise data, except a marginally lower reply prediction PR-AUC with entity-based taskspecific models. Comparing the Avocado and enterprise results, we can see that the performance on all tasks is much better on enterprise users. Our hypothesis is that the larger contact vocabulary in enterprise (1,431 contacts per user on average) compared to Avocado (average 210 contacts per user) makes sender and recipient tasks easier: the distractors are sampled from a larger pool of contacts, and therefore less likely to be related and easier to screen out. In the case of reply prediction, we believe the higher PR-AUC stems from enterprise users that receive a higher volume of machine-generated emails, which have more predictable reply behavior. 5.3
Each set of user-conditioned representations is trained on much
fewer data than most representation learning techniques, but
personalization is a powerful source of context. While our primary
reason for focusing on user-conditioned entity representations is
to avoid privacy leaks, we want to know how they compare against
2Reply prediction is dificult to evaluate in an unsupervised setting; hence, it is not
reported here.
3Our results for sender and recipient prediction through an unsupervised task-agnostic
representation are in the same range as those reported by Graus et al., [
Our primary focus has been unsupervised entity representation
computation. An alternative approach is to induce representations
in a multitask learning setting [
On Avocado, the best multitask model achieves significantly better accuracy in sender and recipient prediction than Pre-trained and Pre-trained + Contacts methods; entity-based task-specific methods are still competitive on recall. We observe the same trend with enterprise, where multi-task models outperform task-specific Pre-trained + Contacts, though entity-based task-specific models outperform multi-task on all tasks and metrics except reply prediction PR-AUC.
Thus the unsupervised methods presented here are competitive with multitask learning on recall despite the fact that they are trained without knowledge of the downstream tasks, and the taskspecific entity-based models are competitive with the multi-task method on accuracy and better on recall. 5.5
To explore the impact of data density, Figure 4 plots sender prediction accuracy using TF-IDF + NMF Centroid representations against the number of emails in a user’s mailbox. Accuracy does not vary substantially, though average recall improves: additional data benefits representing entities on average. Similar trends for other tasks and other models were observed.
We plot the efect of varying dimension sizes for all tasks using the TF-IDF + NMF Centroid approach in Figure 5 for Avocado users. Representations of dimension 400 and 500 consistently achieve best results for both accuracy and recall.
Method
Our current implementation has several optimizations intended for
a production environment. We maintain updates to the TF-IDF
matrix in a streaming manner upon receipt of each email. A periodic
task, run every few days to every week, computes a fresh NMF
representation, using approximately one minute of computation time
per user with an optimized implementation based on Sparse BLAS
operations in Intel MKL. [
We have demonstrated approaches for learning task-agnostic userconditioned embeddings that outperform strong baselines and demonstrate value in a range of downstream tasks. User-conditioned approaches are privacy preserving and show substantial benefits over global models, despite their lower data density. These promising results suggest a range of future directions to explore. One clear next step is to extend our approach to include documents, meetings, and other enterprise entities. Beyond that, embedding relationships between entities could help in predicting more complex connections between them. Next, our explorations in multitask modeling suggest that generalization across tasks also has value. Evaluating the impact of multitask representations on new tasks through leave-one-out experiments may help quantify this.