In the area of ad-targeting, predicting user responses is essential for many applications such as Real-Time Bidding (RTB). Many of the features available in this domain are sparse categorical features. This presents a challenge especially when the user responses to be predicted are rare, because each feature will only have very few positive examples. Recently, neural embedding techniques such as word2vec which learn distributed representations of words using occurrence statistics in the corpus have been shown to be e ective in many Natural Language Processing tasks. In this paper, we use real-world data set to show that a similar technique can be used to learn distributed representations of features from users web history, and that such representations can be used to improve the accuracy of commonly used models for predicting rare user responses.
Predicting the probability of user response such as click, conversion etc. given
an ad-impression is crucial for many advertisement applications, such as
RealTime Bidding (RTB). Because of its e ciency, linear models such as logistic
regression are the most widely used for this purpose[
A similar issue has been recognized in Natural Language Processing (NLP)[
In this paper, we use real-world data set to show that a similar technique can be applied to user response prediction in RTB. Similar to the situation in Natural Language Processing, a large amount of user web history can be used to learn high quality feature representations, which can then be used to predict (rare) user responses. The technique was shown to improve the accuracy of commonly used models, especially when labeled data was scarce. 2
Various methods have been employed to address the feature sparsity problem.
For example, higher order category information derived from human annotation,
or from the data via unsupervised methods such as topic modelling, clustering
etc.[
Another category of solutions involve embedding sparse categorical features
into low-dimensional vector space. Various feature transformation methods that
yield dense features has been investigated in conjunction with deep neural
networks resulting in improvement over major state-of-the-art models[
In this paper, we report initial results of applying a feature transformation
technique similar to neural word embedding to user response prediction in RTB.
The technique has been successfully applied to other domains, such as product
recommendation [
We rst provide a brief overview of the neural word embedding technique
developed by Mikolov et al[
evt vc
where vt and vc 2 Rn are vector representations for t and c, and C is the set
of all available contexts. n is a hyper-parameter that determines the size of the
embedding, and is chosen empirically. Note that we use a distinct representation
for target and context, following the literature. This objective is straightforward
but expensive to calculate. To alleviate this problem, a technique called negative
sampling[
X log (t;c)2D
1 1 + e vt vc +
X (t;c)2D0 log(
1 1 + evt vc ) where D is the set of all target-context pair in the corpus and D0 are randomly generated (t; c) pairs. The objective is now cheap to calculate.
In this paper, we consider a dataset consisting of ad impressions. When an
ad is shown to a user, some of the browsing history of that user is available as
sequence of content IDs. It is thus relatively straightforward to apply techniques
such as CBOW[
We used a real-world RTB dataset provided by Adform. Each record in the
data corresponds to an ad-impression, and is ordered chronologically. The record
consists of a binary label that indicates whether the user subsequently clicked
the ad (click), and a set of content IDs (content_ids) the user had consumed
in the past 30 days, up to the time of the impression. The data was taken from
Adform's impression logs of July 2016. Records for which no content_ids were
available were ltered out. Further, negative examples were down-sampled at
a rate of 0.01 as the data is extremely imbalanced. After the down-sampling,
there were 5.0M examples in total, with 1.1M positive examples. There were
891K distinct content IDs. A newer, larger version of the dataset with additional
elds has been published [
The experiment consisted of an unsupervised stage and a supervised stage.
Unsupervised stage. Content embeddings were learned from content_ids as
described above. I.e. the click eld was discarded and not used for this stage.
Out of the 5.0M data instances, the oldest 4.0M were used for this stage. We
trained the embeddings with varying embedding sizes n (2k2[1: :7]). Tensor ow[
Supervised stage. In the supervised stage, binary classi ers that predicts
click were trained, using di erent features (see below). For all experiments,
Logistic Regression with L2 normalization was used. Out of the remaining 1.0M
data instances, the newest 30% (300K) were held-out as validation dataset. The
training was done with varying amounts of data (0.3K, 1K, 10K, 100K) that were
randomly sampled from the remaining data (700K). To evaluate the performance
of the models, area under the ROC curve (AUC) was used, which is a commonly
used metric for evaluating user response prediction models in RTB[
DR: Distributed Representation. Each dimension of the resulting embeddings were scaled by its maximum absolute value. For each content_id in content_ids, the corresponding embedding was looked up and the mean of the embeddings were used as the feature vector. The resulting feature vector had thus the same length n as the embeddings.
SB+DR: Sparse Binary and Distributed Representation. The feature vector of
SB and DR were concatenated. 4.3
Table 1 shows the best results obtained for each condition using the aforementioned grid-search. The results of SB+DR and DR is compared against SB (our baseline). DR outperforms SB when training data is scarce. SB+DR outperforms SB in all conditions, especially stronger when training data is scarcer. This is likely because when training data is scarce, the sparsity issue is more acute and thus the ability to generalize across features has a larger e ect. However, when large amount of data is available, the lower-dimensional feature representation of DR likely limits the degree of di erentiation between individual content IDs. When SB and DR is concatenated, both advantages can be preserved.
Figure 1 shows the di erence in AUC from the SB baseline for DR and SB+DR, for varying embedding sizes (n). Increasing n improves AUC, but the return diminishes after about 16 dimensions. In this paper, we reported initial results of applying neural feature embedding technique to user response prediction in RTB, using a real-world dataset. To our best knowledge, this is the rst time this technique was applied to this problem. We have demonstrated that the technique can improve performance of commonly used model in the industry, especially when labeled data is scarce and when thus the feature sparsity problem is most acute. The fact that large amount of data can readily be used for training of the feature embeddings, and that the commonly used logistic regression can be used at prediction time make the result ideal for industrial implementation.
The result also opens up exciting opportunities to apply improvements and
techniques that have been developed around neural word embeddings, such as
incorporating global context, using multiple representations per word[