In this short paper, we describe a new model for learning content-based tweet embeddings that serve to be generically useful as signals for a variety of down-stream predictive tasks. In contrast to prior approaches that only leverage cues from the raw text, we take a holistic approach and propose Twice, a model for learning tweet embeddings that (a) leverages cues beyond the raw text (including media) and (b) attempts to yield representations optimized for overall similarity, a combination of topical, semantic, and engagement similarity. Ofline evaluations suggest that our model yields richer and superior embeddings compared to the benchmark models in the tasks evaluating on both academia dataset and twitter products.
be close in the embedding space. An example pair of A rich representation of a tweet that captures nuances semantically similar sentences is “The quick brown fox in meaning is critical for most predictive models at Twit- jumped over the lazy dog” and “The brown fox leaped ter (including Topics, Health, Recommendations etc.). over the lazy dog”. (b) Topic Similarity – tweets that are Consequently, there is an urgent need for models that about the same topic should be close in the embedding can encode or summarize a tweet’s content into a dense space. An example pair here would be “Don Bradman is representation – a representation that can then be used the greatest cricket player of all time” and “India won the in various downstream models. Some key surface areas Cricket World Cup” since both tweets are about “Cricket”. which will be using tweet embeddings include Home (c) Engagement Similarity – tweets that share engagement Timeline, Notifications, Topics, and potentially Health audiences are deemed similar. models. An important requirement is that the tweet em- In this paper, we present Twice – a model that atbedding be generically useful on a variety of predictive tempts to capture the above notions of tweet similarity. tasks and not necessarily be useful for only a specific In contrast to most prior work which only seeks to emapriori task. Finally, we expect downstream models to bed raw tweet text using standard pre-trained language only consume the embeddings, without having to worry models, Twice models tweets holistically leveraging not about the inner workings of the underlying model. only raw tweet text, but also incorporating cues from the
Taking a bird’s eye view, the need is simply for a associated media, and hyperlinks. We evaluate Twice on tweet representation/embedding that captures similarity a suite of ofline benchmark tasks and demonstrate that between tweets by embedding them in a vector space. our proposed model significantly outperforms several Specifically, tweets that are “similar” must be close in the baseline approaches. vector space and tweets that are not “similar” should ideally be far in this vector space with respect to a suitable metric. Zooming in, for practical modeling it is useful to 2. Related Work attempt to operationalize this vague notion of “similar” and attempt to be more specific here. We can attempt to capture the following notions of similarity: (a) Semantic
of learning sentence embeddings which seeks to learn
dense representations of sentences and capture sentence
similarity. One of the earliest works on learning dense
dDaLt4ioSnR,’2c2o:-lWocoartkesdhowpitohnthDeee3p1sLteAarCnMingInftoerrnSaeatirocnhaalnCdonRfeecroemncme eonn- embeddings of sentences is the work of Le and Mikolov
Information and Knowledge Management (CIKM), October 17-21, 2022, [
At its core, Twice is a Bert model [
The full loss function is simply the average of the above
three loss functions. Finally, to obtain a dense
representation of the tweet, we simply use the representation of
the [CLS] token. Twice leverages cues from the entire
tweet and not just the raw tweet text. In particular, in
addition to the raw text, we also leverage media cues by
obtaining media annotations for any associated media as
described in [
Setup. We quantify the efectiveness of tweet
embeddings in capturing content similarity via measuring their
performance on three benchmark tasks – tasks that
relfect how well embeddings capture notions of similarity
noted in Section 1:
• SemEvalPIT [
Our rationale for restricting ourselves to using very simple nearest neighbor approach based on cosine similarity of tweet embeddings is based on the intuition that higher quality representations would inherently demonstrate a higher-degree of “ease of extraction” of the predictive information. It is for this reason precisely that one needs to use simple models as opposed to very complex deep predictive models. We made a design choice to use a NN-based approach which supported quick implementation but simple shallow models are another alternative. Finally, we summarize model performance by reporting the harmonic mean over tasks.
Baselines.
We consider the following baseline models:
• Bert: This is just the standard pre-trained Bert
model [
Simcse Hashspace Topicspace
Clip Twice
Results. Table 1 shows the results of our evaluation on using the Twice embeddings on the tasks of Spam deour benchmark suite. Based on these results we can make tection and ToS violation classification. We compare the the following conclusions: (a) First, observe that just us- result with the benchmark models Bert and Hashspace. ing standard models like Bert for embedding tweets does The results show that Twice outperforms both the stannot yield superior embeddings. It is imperative to learn dard Bert and Hashspace models in both the Spam deembeddings from Twitter data. (b) Second, state-of-the- tection and ToS violation tasks. art unsupervised methods for sentence embedding perform worse than supervised methods which is inline with 4.3. Qualitative Evaluation and Analysis prior work on sentence embeddings as well. (c) Topic- of Usage in Content Recommenders space significantly outperforms Hashspace overall. This is because Topicspace leverages cues from beyond tweet In order to evaluate our model qualitatively, we also built text and also uses a simpler but more intuitive task. (d) a web page where one can enter a tweet ID and see the Models like Topicspace and Clip which solely optimize nearest neighbors to the given tweet from a given prefor a specific notion of similarity tend to perform signifi- determined universe of tweets – a scenario that reflects cantly better on the corresponding evaluation tasks than the usage of embeddings for candidate generation in conother models simply because the underlying represen- tent recommenders. Figure 2 shows the nearest neighbors tations are optimized to capture that specific similarity for a couple of seed tweets as a demonstration. Note that notion over others. (e) Finally, note that our proposed the nearest neighbors reflect the broad topic of the seed model Twice generally outperforms all of these base- tweet and are similar in content to the seed tweet suggestline approaches overall. While the mean performance of ing that our model is able to capture content similarity Twice and Clip are identical, note that Twice outper- between tweets. forms Clip on both SemEvalPIT and the Topics tasks While the process outlined in this section can be used significantly with a slight drop on the Favs task. for candidate generation in content recommenders (by
To summarize, all in all Twice demonstrates superior ifnding tweets similar to a user’s interests and past enperformance over prior production models and yields im- gagements), through a qualitative analysis conducted proved embeddings of tweets by leveraging cues beyond ofline, we have identified a list of challenges that need to the tweet text. be addressed in-order to ensure good quality candidates are returned.
Setup. We evaluate the performance of content embeddings on our health platform. We use Twice embeddings as features in a shallow model to predict Spam and Terms of Service (ToS) violations. The spam prediction is a binary classification task to predict whether a tweet is a • Seed Selection. Using tweets with which users have positively engaged as seeds to fetch more interesting content is a natural choice. However, some seed tweets may have very little content which makes them unsuitable for retrieving candidate tweets. Examples include tweets that contain frequent phrases like “good morning”, everyday greetings, and daily life updates.
Bert Hashspace
Twice
Note that these challenges are independent of the underlying tweet representation itself and may significantly hinder the quality of candidates even if the tweet representation model is of a superior quality. To address these challenges, we build various filters and apply to the candidate pool, we also carefully design and pick desirable seed tweets to generate high quality tweets for the user.
In this paper, we proposed a model for embedding tweets that goes beyond just modeling tweet text. Our goal has been to develop generically useful rich representations of tweets that can be used in a variety of downstream predictive models at Twitter. To that end, we have demonstrated through ofline evaluation that our proposed model outperforms the benchmark models on various tweet products. As next steps, we seek to validate our model using online A/B tests in various product surfaces which serve as the ultimate litmus test.