The study of collective attention is of growing interest in an age where mass- and social media generate massive amounts of often short lived information. That is, the problem of understanding how particular ideas, news items, or memes grow and decline in popularity has become a central problem of the information age. Recent research e orts in this regard have mainly addressed methods and models which quantify the success of such memes and track their behavior over time. Surprisingly, however, the aggregate behavior of users over various news and social media platforms where this content originates has large been ignored even though the success of memes and messages is linked to the way users interact with web platforms. In this paper, we therefore present a novel framework that allows for studying the shifts of attention of whole populations related to websites or blogs. The framework is an extension of the Gaussian process methodology, where we incorporate regularization methods that improve prediction and model input dependent noise. We provide comparisons with traditional Gaussian process and show improved results. Our study in a real world data set, uncovers hidden patterns of user behavior.
Over the last couple of years, so called \question answering" (QA) sites have
gained considerable popularity. These are internet platforms where users pose
questions to a general population. Yahoo Answers, Quora and the Stack
Exchange family establish internet communities which provide natural and seamless
ways for organizing and providing knowledge [
Due to the complexity of user-system interaction (millions of people discuss
thousands of topics), exible and accurate models are required in order to
guarantee reliable forecasting. In recent years the Bayesian setting and the Gaussian
Process (GP) framework [
In the next section, we formally introduce the Gaussian Process framework and provide details regarding our extensions towards variable noise models. We then show an analysis of the periodicity of the time series of tag activity as apparent from the Stackover ow data set. Next, we compare our prediction results with those of other models and discuss the advantages of introducing functional dependencies on noise terms. Finally, we provide conclusions and directions of future work. 2
In this section, we propose a Gaussian process (GP) model for regression that
extends the sparse pseudo-input Gaussian process (SPGP) for regression [
A GP is a Bayesian model that is commonly used for regression tasks [
Next, we rst brie y review the GP model for regression, yet, for a detailed
discussion we refer to [
N Consider a data set D of size N containing pairs of input vectors X = fxngn=1
N and real value target points y = fyngn=1. In order to apply the GP model to regression problems, we need to account for noise in the available target values, which are thus expressed as
yn = fn + n where fn = f (xn) and n is a random noise variable which is independently chosen for each n. We shall consider a noise process following a Gaussian distribution de ned as p (y j f ) = N y j f ; 2I where N (y j m; C) is a Gaussian distribution with mean m and covariance C. The marginal distribution of p (f ) is then given by another Gaussian distribution, namely p (f j X; ) = N (f j 0; KN ). The covariance function that determines KN is chosen to express the property that, if points xn and xm are similar, the value [KN ]nm should express this similarity. Usually, this property of the covariance function is controlled by small number of hyperparameters . Integrating out over f , we obtain the marginal likelihood as
Z p (y j X; ) =
p (y j f ) p (f j X; )
where [kx]n = K (xn; x) and Kxx = K (x; x). In order to predict with GP model,
we need to have all the training data available during run-time, which is why
the GP for regression is referred to as a non-parametric model.
An approximation of the full GP model for regression is presented in [
Z p y j x ; D; X =
p y j x ; X; f p f j D; X df where KN is the covariance matrix of the training data, KM is the covariance matrix of the pseudo inputs, 2 is the noise, Q is de ned as and is de ned as
Q = KMN + 2 1
KNM = diag KN
KNM KM1KMN : Finding the pseudo input locations X and the hyperparameters (kernel parameters and noise) = f ; 2g can be done by maximizing the marginal likelihood (8) with respect to the parameters fX; g.
p y j X; X;
Z
=
One positive e ect of the sparsity of the SPGP model is the capability of
learning data sets that have variable noise where the term variable noise refers to
noise which depends on the input. However, it is important to note, that this
capability is limited and an improvement of the SPGP model is presented in
(5)
(6)
(7)
(8)
[
KM ! KM + diag (h) ;
(9)
where h is a positive vector of uncertainties that needs to be learned and diag (h)
represents a diagonal matrix whose elements are those of the h vector. This
extension allows the possibility of gradual in uence on the pseudo inputs. This
means that if uncertainty hm = 0, then the pseudo input m behaves like in the
standard SPGP. Yet, as hm grows the particular pseudo input has less in uence
on the predictive distribution. This possibility of partially turning o the pseudo
inputs allows a larger noise variance in the prediction. The authors of [
Introducing the heteroscedastic extension to the SPGP empowers the model to learn from data sets with varying noise. However, making the model this exible may cause problems of over tting. Also, using the SPGP+HS to predict user and website activities, does not allow us to interpret the behavior of the noise because noise is represented as a positive vector h of uncertainties and attempts of interpreting these values do not yield meaningful information about the behavior of the noise.
One way of solving the problems of over tting and lack of interpretability will be to put a prior distribution over the vector h of uncertainties. However, taking this approach leads to computationally intractable integrals.
The solution which we propose for these problems is to make use of an uncertainty function that depends on the pseudo-inputs. Our covariance function of the pseudo-inputs is de ned as
KM ! KM + diag (fh (xm)) ; (10) where fh is the uncertainty function and xm is a pseudo-input. By de ning the heteroscedastic extension in this way, it is possible for the parameters of the uncertainty function to be learned by the gradient based maximum likelihood approaches. Hence, later on, we are able to interpret the parameters of the heteroscedastic noise function as parameters that govern the noise in the model. Another advantage of having a heteroscedastic function is that it restricts the parameter search space when learning the model. This restriction can be benecial when learning the model, because, it removes unnecessary local maxima. This results in much faster convergence when learning the model and also in improved chances of reducing over tting. In the following, we will refer to our new heteroscedastic function model as SPGP+FUNC-HS.
For modeling the Stackover ow data set, we introduce two heteroscedastic noise functions. In general, we may use any function that can describe the noise of the given data set. The rst heteroscedastic noise function which we consider is the simple sine function de ned by
fh (xm) = a sin (2 !xm + ') , where a is the amplitude, ! is the frequency and ' is the phase. We refer to this model as SPGP+SIN-HS. The second heteroscedastic noise function we investigate is a product of the sine function and an RBF kernel, namely fh (xm; hm) = c2e (xm2lh2m)2 sin (2 !xm + '), (11) (12) where c is the variance, hm is a mean associated with every pseudo-input xm in the RBF kernel, and l is the length scale of the RBF kernel. The mean in the RBF kernel can be initialized at random or set by the user if the user has corresponding prior knowledge. Setting a mean for every pseudo-input point divides the whole input space into regions where, in each region, we have a function governing the uncertainty associated with every pseudo input. The uncertainty function de ned like this then behaves like mixture of experts and we refer to this model as SPGP+RBFSIN-HS model. 3
1010 ]z109 H / **2108 V [D107 S P106 105
Time P1e01riods Log(Days) 102
In the previous section, we presented the Gaussian process method and two extensions of this method, the SPGP+HS and the SPGP+FUNC-HS. In this section, we present results we obtained when using these models on our Stackover ow data set.
In order to test our models, we used publicly available data-dumps of Stackoverow1. The data set contains the number of questions and answers of postings 1 Downloadable URL: www.archive.org/details/stackexchange classi ed by tag for every business day. The models are trained on a data set containing information about daily postings in the time between 01.02.2014 to 31.08.2014. The evaluation of the models is done on a test set containing postings for the rst 21 working days in September 2014.
The performance of the presented models depends on the choice of the kernels
used for the covariance matrix. When working with GPs, an additional analysis
is required to select proper kernels for the covariance matrix. Because we work
with a data set that re ects user behavior, we supposed that it may show a form
of periodicity in the behavior of the users. Accordingly, we performed spectral
density estimation analysis [
A periodogram of the time series data that we are analyzing is shown in Figure 1. Since all our tag related time series tag have almost the same periodogram, we only show one of them. For better interpretability we converted the frequencies into periods to observe in how many days the periods occur. There are two apparent peaks, the rst occurring at two and a half days and the second at ve days. In this case the period of ve days appears as an echo of the two and a half days period, therefore we dismiss the second period and we only take into account the rst period. Additional characteristics of this data set are minor irregularities and a long term rising trend in the overall time series. Given these observations, our models that show the best performance as a covariance function use a sum of four kernels k (x; x0) = k1 (x; x0) + k2 (x; x0) + k3 (x; x0) + k4 (x; x0) : (13) The question of how to choose these kernels and the particular role of each kernel in the learned model will be discussed in the next section.
Next, we present the result achieved for the top ten tags according to the number
of posted questions and answers in the 2014 Stackover ow data set. Table 1
presents the results of the di erent models of the posted questions time series and
Table 2 presents the results of the di erent models for the posted answers time
series. In order to compare the prediction models, we considered the following
measures:
{ Mean Square Error (MSE) to accounts for the accuracy of the prediction
of an unseen data point
{ Negative Log Predictive Distribution (NLPD) to obtain a con dence
for the predicted values on an unseen data point
{ Negative Log Marginal Likelihood (NLML) to account for how well
the model ts the training data.
For the MSE and the NLPD measures, smaller values are better, and for the
NLML larger values are better. The best model for each tag has been chosen
using the Akaike information criterion (AIC) [
In Fig. 2, we present two learned models, one for the tag \Java" (Fig. 2a) and one for the tag \iOS" (Fig. 2b). We observe that the model that models the Java tag strives to predict the test point using the mean. In contrast, the model that models the iOS tag predicts the test points in terms of noise.
training data test data mean The di erent kernels in Eq. (13) allow us to dissect the dynamical behavior of the population w.r.t. di erent scales and patterns. In order to portrait these behaviors, we calculated the mean function and variance Eq. (5) by generating vector k> using independent kernels. We present the values of each kernel in the \android" question data set in Fig. 3 { Mean trends (Fig. 3a) characterize the behavior of the population of users over scales measured in months and represent the global mean behavior of the population. We hypothesize that they are driven by the shear size of the user base. The more people interested in the tag are visiting the site, the higher the average number of questions per month. Further, this overall trend might represent the changes in the dominance of this particular tag of questions in the data set. Because the tag refers to a programming language, trends like this indicate changes in attention to various languages. Such dynamics are modeled using the rational quadratic kernel In this paper, we addressed the problem of forecasting the daily posting behavior of users of the Stackover ow question answering web platform. In order to accomplish this task, we extended the variable noise pseudo inputs Gaussian training data mean ya 100 repd 50 iitss 0 v fo# 50 100
Feb Mar
Process framework by introducing a functional noise variant. The idea of using functional descriptions of noise allowed us to study periodic patterns in collective attention shifts and was found to act as a regularizer in model training. Our extended Gaussian Process framework with functional representations of various kinds of noise provides the added advantage of increased interpretability of results as the di erent kernels de ned for this purpose can uncover di erent kinds of dynamics. In particular, our kernels revealed major distinct characteristics of the question answering behavior of users. First of all, there are major trends on time scales of about six months showing growing and declining interest in particular topics or corresponding tags. Second of all, these major trends are perturbed by seasonal behavior, for example overall activities usually drop during the summer season. Third of all, on a ne grained scale, there are weekly patterns characterized by periods of 2.5 days. Fourth of all, there are noisy uctuations in activities on daily scales.
Given the models and results presented in this paper, there various directions for future work. First and foremost, we are currently working on implementing a distributed Gaussian Process framework in order to extend our approach towards massive amounts of behavioral data (use of tags, comments, and likes) that can be retrieved from similar social media platforms such as Twitter or Facebook.