With the rapid development of mobile devices, spatial location-based crowdsourcing applications have attracted much attention. These applications also introduce new security risks due to untrustworthy data sources. In the context of crowdsourcing applications for spatial interpolation (i.e. spatial regression) using crowdsourced data, the results can be seriously affected if malicious data sources initiate a colluding (collaborate) attacks which purposely alter some of the measurements. To combat this serious detrimental effect, and to mitigate such attacks, we develop a robust version via a Gaussian Process mixture model and develop a computationally efficient algorithm which utilises a Markov chain Monte Carlo (MCMC)-based methodology to produce an accurate predictive inference in the presence of collusion attacks. The algorithm is fully Bayesian and produces posterior predictive distribution for any point-of-interest in the input space. It also assesses the trustworthiness of each worker, i.e. the probability of each worker being honest (trustworthy). Simulation results demonstrate the accuracy of this algorithm.
Crowdsourcing, as defined by Howe [1], is often characterized by the participation of large networks of people from the public in tasks given by companies or institutes. It has been broadly used in data acquisition, which replaces the traditional data gathering processes that are costly. Recent works have shown many successful applications in crowdsourced information gathering from various different domains. In [2], the authors introduce a crowdsourcing framework for acquisition and analysis of mobile videos to monitor rapidly changing disaster sites that optimally utilise bandwidth and resources. In a study done by Meier et al. [3], crowdsourced atmospheric data in Berlin, Germany was used for urban climate research. In [4], a privacy-preserving community sensing method is developed for real-time road traffic monitoring. Another study by Kim et al. [5] uses crowdsourcing and regression techniques to automatically analyse the conflict level in television political debates.
While decentralized spatial information gathering (crowdsourcing) techniques have clear benefits, such as being inexpensive and easy to implement, their quality and reliability are not as good as data collected from domain experts. The issue of trust is one of the major impediments to the success of real-world crowdsourcing applications [6]. Apart from inherent noises, untrustworthiness in crowdsourced data can originate from various ways. For example the sensor used for measurement can be faulty (if the data is in the form of sensor reading). Data that come from humans can be subjective and highly biased [7]. For example, workers might put very little effort into the work they are assigned in the hope that they will not be noticed and still get the reward. Other workers might lack the essential skills to perform the task, hence produce erroneous data [7]. In addition, there can be adversaries that report malicious data strategically to interrupt the crowdsourcing application [8].
The challenge of identifying and filtering untrustworthy information before summarization (prediction, decision making, etc.) has been addressed by a number of researchers in various domains. Some of these works focus on binary/multi-class labelling tasks. Groot et at. [9] applied a Gaussian Process for regression model to merge information from multiple inaccurate annotators to produce reliable labels for objects. Tarasov et at. [10] proposed a method to dynamically estimate reliability of workers in crowdsourcing for regression by adopting a multi-armed bandits (MAB) model.
Some studies are applied to problems of estimation. Reece et al. [11] studied the problem of tackling sensor faults in applications based on sensor networks. Their method consists of two stages. Firstly the model-based step detects and filters sensor faults with known types, then the consensus-based step removes unanticipated faults that passed the first step. Venanzi et al. [12] developed an algorithm called MaxTrust that jointly estimates the target attribute and the trustworthiness of each worker using maximum likelihood paradigm.
Not much attention has been given to address the problem of spatial regression (or the estimation of a spatial process/function) with the presence of untrustworthy data sources. Venanzi et al. [13] proposed a heteroskedastic Gaussian Process (HGP) model with independent noises and designed an algorithm called TrustHGP to jointly estimate the spatial function and the trustworthiness of each worker. Their model assumes that all reports are noisy observations from the same underlying function with heterogeneous and independent noises. Malicious observations are associated with higher noises compared to honest observations. These assumptions of their model are too intuitive to be able to handle complex real-world situations. Especially, this method is vulnerable against collusion attacks.
In collusion attack scenarios in crowdsourcing, multiple untrustworthy workers can collaborate to inject deceptive false data to skew the regression results. More specifically, an effective strategy for attackers is to make a number of similar reports that are far from the ground-truth (or follow a spatial function that is far from the underlying spatial function) with similar input values. This often results in the regression algorithm mistakenly accepting the false value provided by attackers. We refer to this type of attacks and similar types as collusion attacks. We will formally present the attacker model in the model specification in Section 3 of this paper. 1.1
In order to address the issue of collusion attacks, we propose a Gaussian Processes mixture model where malicious data and non-malicious (honest) data are modelled to be from two different realizations of Gaussian Processes. We develop a Markov chain Monte Carlo (MCMC) based algorithm that takes all data as input and produces statistical prediction of function value at any spatial location of interest. The predictions produced by the algorithm is based on honest data. It minimizes the impact of malicious data by excluding them before performing posterior inference with high accuracy. The algorithm also assesses the trustworthiness of each worker in terms of the probability that the data it produces being honest. We show that this method is truly collusion-resistant by performing experiment on synthetic dataset. 1.2
We develop a unified statistical framework for collusion attacks in the regression setting via a Gaussian Process mixture model.
We develop a novel MCMC-based algorithm to produce comprehensive summary of data in terms of posterior predictive distribution and trustworthiness of each worker (data point). The inference method is fully Bayesian.
We conduct a proof of concept on synthetic dataset and show that our method is truly resistant to collusion attacks. In the field of spatial monitoring, n sets of data are gathered from n different workers that are potentially untrustworthy. Each set of data comes in the form of a tuple (xi; yi); i = 1; : : : ; n, where xi 2 Rd represents the independent variables (spatial location, time, input features, etc.), and yi 2 R represents the response (dependent) variable that depends on xi. Let x := fx1; : : : ; xng, and let y := fy1; : : : ; yng. We refer to a tuple of (xi; yi) as a data point. Each data point is either honest or malicious, depending on whether it comes from a trustworthy or untrustworthy worker. Honest data points are noisy observations from the underlying function fH : Rd ! R, while malicious data points are not (and will be defined later).
The task of the regression problem is that given x and y, produce a reliable estimation of fH(x ) with its probability distribution, for any given x 2 Rd, that is free from the influence of malicious data. The probability density function p(fH(x )jx ; x; y) is referred to as posterior predictive distribution and contains all the information about fH(x ), given all the observations.
To motivate our model, we present a real-world example of the stated problem. Suppose a meteorological study aims to estimate the spatial structure of atmospheric parameters (e.g. temperature, humidity, air quality, etc.) of a particular region of interest by using crowdsourced data collected from workers in the region by hand-held devices equipped with sensors. In this example, each report (xi; yi) consists of: xi, the geographical location of the i-th worker (longitude and latitude), and yi, the reading of the atmospheric parameter from the sensor yi, e.g. air quality. The underlying function of interest fH corresponds to the inherent spatial structure of the parameter, i.e. for a given location x , fH(x ) corresponds to the ground-truth of the value of the parameter at the location. Suppose that, for some reasons, a worker reports dishonest data at location xm that does not correspond to the actual parameter value fH(xm). This piece of data is deemed as malicious. There can be practical reasons why malicious data exists, such as to defame a spatial location by claiming that the air quality in the vicinity is poor. There can be multiple malicious workers reporting dishonestly to achieve their objectives. Thus, the data reported by these workers cannot be trusted and should be regarded as unrelated to fH. 3
In this section, we formally present the statistical crowdsourcing model which uses Gaussian Process mixture model.
We model the physical phenomena as spatially dependent continuous processes with a spatial correlation structure and are independent from each other. The degree of the spatial correlation in the process increases with the decrease of the separation between two observing locations and can be accurately modelled as a Gaussian random field 1. A Gaussian process (GP) defines a distribution over a space of functions and it is completely specified by the equivalent of sufficient statistics for such a process, and is formally defined as follows. Definition 1 (Gaussian process [14]): Let X RD be some bounded domain of a D-dimensional real valued vector space. Denote by f (x) : X 7! R a stochastic process parametrized by x 2 X . Then, the random function f (x) is a Gaussian process if all its finite dimensional distributions are Gaussian, where for any m 2 N, the random variables (f (x1) ; ; f (xm)) are normally distributed.
We can therefore interpret a GP as formally defined by the following class of random functions:
F := ff ( ) : X 7! R s.t. f ( ) (x; ) := E [f (x)] : X 7! R;
GP ( ( ; ) ; C ( ; ; )) ; with C (xi; xj; ) := E [(f (xi) (xi; )) (f (xj) (xj; ))] : X
X 7! R+ ; where at each point the mean of the function is ( ; ); parameterised by , and the spatial dependence between any two points is given by the covariance function (Mercer kernel) C ( ; ; ), parameterised by (see detailed discussion in [14]).
1We use Gaussian Process and Gaussian random field interchangeably.
We assume that fH can be accurately modelled a Gaussian Process prior with zero mean and square exponential covariance function parameterised by H2; wH21; ; wH2d.
Similarly, fM is also modelled a Gaussian Process prior with zero mean and square exponential covariance function parameterised by M2; wM21; ; wM2d.
CH(xi; xi0 ) = H2 exp
d 1 X (xim 2 m=1 wH2m
xi0m)2 ! fH
GP(0; CH); fM
GP(0; CM); CM(xi; xi0 ) = M2 exp
d 1 X (xim 2 m=1 wM2m xi0m)2 ! : :
Data points are partitioned into 2 mutually exclusive sets, H and M, denoting honest data points and malicious data points. In other words, either i 2 H or i 2 M. Without prior knowledge, H is indistinguishable from M. Hence, we make the assumption that H contains the majority of data points, that is, jHj > jMj. The case where malicious data points become the majority is unrealistic in real applications. When it happens, there is no way to distinguish which set is honest without informative prior knowledge.
Since each data point belongs either to H or M, we use c 2 fH; Mgn to denote the configuration vector, where each component ci denotes the set which i belongs to. In other words,
As mentioned above, honest data points are noisy observations from underlying function fH : Rd ! R. Similarly, malicious data points are noisy observations from a different function fM : Rd ! R. Hence, ci = (H if i 2 H
:
M if i 2 M yi = (fH(xi) + i; if ci = H fM(xi) + !i; if ci = M
;
i iid N (0; n2); !i iid N (0; !2):
(
Let denote hyperparameters of above GPs (including variances of noise), = ( H2; M2; wH21; ; wH2d; wM21; ; wM2d; n2; !2):
In these hyperparameters, signal variances H2 and M2, noise variances n2 and !2 are given inverse-gamma priors, 2 ; M2 iid Inv-Gamma( f ; f ); n2; !2 iid Inv-Gamma( n; n);
H and length-scales wH21; ; wH2d; wM21;
; wM2d are given log-normal prior, wH21; ; wH2d; wM21;
; wM2d iid ln N ( w; w2):
Placing priors over hyperparameters makes the method fully Bayesian and provides extra flexibility to the model. Parameters f ; f ; n; n; w; w2 are pre-set values to make these priors non-informative. Alternatively, when prior knowledge about hyperparameters is available, these parameters can be tuned in order to achieve better accuracy.
In addition, we define the following notations that are used in the following parts of the paper. xH = fxi : ci = Hg; xM = fxi : ci = Mg; yH = fyi : ci = Hg; yM = fyi : ci = Mg;
n n nH = jHj = X (ci; H); nM = jMj = X (ci; M): i=1 i=1 3.4
In our assumptions, xH is distributed uniformly throughout the domain of x since no region is preferred over other regions when gathering data. When prior knowledge about distribution of independent variables is present, the uniformity assumption can be replaced with no difficulty.
xHim iid U (lm; um); for i = 1 n; m = 1 d; where xHim denotes the mth dimension of ith honest data point, lm, um are the lower bound and upper bound of the mth dimension.
It is assumed that xM follows a d-variate Gaussian distribution with mean x and covariance x. This is a practical assumption since in the presence of collusion attacks, the damage maximizing strategy for attackers is to concentrate malicious data points in a particular region. In addition, attackers are assumed to possess limited resources, making it difficult for them to inject malicious data from a broad region (for example, when the independent variable is geographical locations). In the case of a distributed attack where independent variables of malicious data spread through a broad region, x corresponds to a large value that makes the distribution of xM flat.
xMi iid N ( x; x); for i = 1 n; where xMi denotes the ith malicious data point.
Let denote parameters of the input distribution, where x is given uniform prior, and
x is given inverse-Wishart prior, and the parameters ; are set to be non-informative. 3.5
= ( x; x); xm
U (lm; um); for m = 1
d; x
W 1( ; ); We assume there is no a priori information about the trustworthiness of each worker. Hence there is an identical probability of each data point being honest. Therefore, c has a n-variate Bernoulli distribution with parameter (H corresponds to the positive outcome and M corresponds to the negative outcome), It is obvious that nH follows a binomial distribution with parameters n and .
ci iid Bernoulli( ); for i = 1
n: nH
B(n; ):
Strictly speaking, since nH > nM, nH should follow a truncated binomial distribution where nH > n2 . For simplicity, this condition is ignored here. As later discussed in the algorithm, if a posterior sample violates the condition, we simply ignore the sample.
is given a beta prior, with known parameters and ,
):
This full posterior p(c; ; ; jx; y) is clearly intractable. However, if we could sample from this posterior
p(c; ; ; jx; y) and get S samples f(c(
S 1 X p(fH(x )jx ; x; y; c(r); (r))
S This posterior predictive distribution is a mixture of multivariate Gaussian distribution, whose mean and covariance matrix fH(x ) are given by: fH(x ) p(c; jx; y) =
p(c; ; ; jx; y)d d ; p(c; ; ; jx; y) / p(yjx; c; )p(xjc; )p(cj )p( )p( )p( ):
S 1 X fH(x ) = S
r=1 fH(x ) =
S X( (r) + r=1 (r); (r) (r)T )
T fH(x ) fH(x );
Based on the model we propose, the goal is to obtain the posterior predictive distribution, which is derived by marginalizing over c and , i.e.
p(fH(x )jx ; x; y) =
p(fH(x )jx ; x; y; c; )p(c; jx; y)d :
(
Therefore, we develop a MCMC sampler algorithm to generate samples from the posterior distribution p(c; ; ; jx; y). We can use Gibbs Sampling to generate samples from conditional posterior distribution of each component of the parameter space. 4.1
Since it is difficult to sample c as a whole from its conditional posterior distribution, we sample it component by component. Let c i denote the vector resulting from removing i-th component from c. p(ci = jjc i; x; y; ; ; ) / p yijci = j; c i; xj i ; yj i ; p(xijxj i ; ci = j; c i; )p(ci = jjc i; ) = N (yi; j ; j )p(xijxj i ; ci = j; c i; )p(ci = jjc i; ); for j 2 fH; Mg; The first term inside the integral p(f (x )jx ; x; y; c; ) is the posterior distribution of Gaussian Process regression and is Gaussian:
p(fH(x )jx ; x; y; c; ) = N (fH(x ); f ; f ); where f = CH(xH; x )T [CH(xH; xH) + n2I] 1yH; f = CH(x ; x ) CH(xH; x )T [CH(xH; xH) + n2I] 1CH(xH; x ): The second term inside the integral p(c; jx; y) is the marginal posterior distribution of c and marginalizing out and from the full posterior p(c; ; ; jx; y).
obtained by X Z c Z 1 Z 0 where p(xijxj i ; ci = H; c i; ) = p(xijxj i ; ci = M; c i; ) = N (xi; x; x); p(ci = Hjc i; ) = ; p(ci = Mjc i; ) = 1
: M = [CM(xi; xi) + !2]
1 d Q (um m=1 lm)
; H = CH(xH i ; xi)T [CH(xH i ; xH i ) + n2I] 1yH i ; H = [CH(xi; xi) + n2]
CH(xH i ; xi)T [CH(xH i ; xH i ) + n2I] 1CH(xH i ; xi); M = CM(xM i ; xi)T [CM(xM i ; xM i ) + !2I] 1yM i ;
CM(xM i ; xi)T [CM(xM i ; xM i ) + !2I] 1CM(xM i ; xi); q(ci 6= cijci) = pc; q(ln H j ) = N (ln H ; ln M; F2 ); q(ln M j ) = N (ln M ; ln H; F2 ); q(ln n j ) = N (ln n ; ln !; N2 ); q(ln ! j ) = N (ln ! ; ln n; N2 ); q(ln wHm j ) = N (ln wHm ; ln wMm; W2 ); for m = 1 q(ln wMm j ) = N (ln wMm ; ln wHm; W2 ); for m = 1 d; d: In above equations, xj i = x n xi, yj i = y n yi for j 2 fH; Mg. Note that rank-one updates to the inverse covariance matrix can be performed to reduce the number of inversion needed.
We can sample the entire from its conditional posterior distribution.
p( jx; y; c; ; ) / p(yjx; c; )p( ) d = p(yjx; c; )p( H2)p( M2)p( n2)p( !2) Y p(wH2m)p(wM2m) m=1 The unnormalized distribution can be derived analytically, along with its partial derivative w.r.t. each parameter. Hence Hamiltonian Monte Carlo (HMC) [19] can be used to efficiently sample from this conditional posterior.
When sampling c and , a convergence issue may occur. Intuitively, the joint posterior distribution
p(c; ; ; jx; y) should be bimodal, since the model specification of c and is highly symmetric except for the
condition that nH > nM, as reflected in the prior of c, and the distribution of independent variables xH and xM.
Thus, switching the labels of all data points (labelling all honest data points as M and all malicious data points
as H) will result in an inferior mode in the posterior distribution (besides the major mode we are interested in
where all data points are correctly classified). This mode has much less probability mass as compared to the
major mode, and since its violation of the condition nH > nM, it will eventually be rejected when producing
posterior predictive inference. Nonetheless, its existence will make the MCMC sampler algorithm highly
sensitive to initial state. Due to the nature of Gibbs samplers, c is updated one component at a time. The slow
movement of the Gibbs sampler in the state space will make it extremely unlikely to jump from a state close
to inferior mode to a state close to the major mode, due to low probability mass between two modes. Hence,
apart from the Gibbs sampler for c and HMC sampler for mentioned above, an auxiliary Metropolis-Hastings
sampler is introduced to facilitate the movement between two modes. The Metropolis-Hastings proposal
distribution will propose a switch of labels as well as their associated GP hyperparameters . Below shows the
proposal distribution q(c jc) and q( j ) of the Metropolis-Hastings jump.
(
F switches labels of all data points along with their corresponding hyperparameters. The update is accepted with probability shown below: p(accept) = min p(c ; ; ; jx; y)q(cjc )q( j ) p(c; ; ; jx; y)q(c jc)q( j ) q( j ) is equal to 1 since the part of the Metropolis-Hastings update related to is symmetrical. Therefore, q( j ) the sampler for c and becomes a mixture of two MCMC samplers. One is the Gibbs sampler for c and HMC sampler for , and the other is a Metropolis-Hastings sampler that jointly updates c and . At each iteration of the MCMC sampler algorithm, with probability pMH, the Metropolis-Hastings sampler is selected. Since the sole purpose of this sampler is to get the chain out of the inferior mode, pMH should be set close to 0. Now when the chain walks into states close to the inferior mode, the Metropolis-Hastings sampler will take the chain to states close to the major mode. When the chain is already close to the major mode, the Metropolis-Hastings jumps will almost never be accepted. To sample , we generate a random sample from the conditional posterior of each component, i.e. from p( xjxM; c; x) and p( xjxM; c; x), which are analytically tractable.
For x, where Z = P (xi x)(xi
fi:ci=Mg and can be directly generated.
x)T : This conditional posterior distribution is an inverse-Wishart distribution 1 C 1 x C x
N N
Q fi:ci=Mg
Q fi:ci=Mg
N ( x; xi; x) Qd 1
m=1 (um lm) N ( x; xi; x) Qd 1
m=1 (um lm) d x x; xM; nMx x; xM; nMx d x
; p( xjxM; c; x) = R =
R
x = W
p(xMjc; x; x)p( x) x p(xMjc; x; x)p( x)d x
! where C = fi:cQi=Mg R x N ( x; xi; x)d x is the product of all the normalizing constants to truncate the Gaussian distribution. Since C appears in both the denominator and the numerator, it cancels out. Note that this conditional posterior distribution is a truncated Gaussian distribution and can be efficiently generated.
For x, we have that
(
Input: S, x, y and parameters of priors
Output: S samples of posterior distribution (c(
Sample x from p( xjx; y; c; ; x; ).
Sample x from p( xjx; y; c; ; x; ).
Sample from p( jx; y; c; ; ).
Save current values of c, , , in (c(r), (r), (r), (r)).
19 return f(c(
, (c(S), (S), (S), (S))g. 4.5
Finally, to sample tion. , (c(S), (S), (S), (S)) we generate a random sample from its conditional posterior, which is also a beta distribup( jx; y; c; ; ) = p( jc) =
p(cj )p( )
R01 p(cj )p( )d
=Beta ( ;
+ nH;
+ n
nH) :
(
Our methods consists of two algorithms. The first, presented in Algorithm 1, is a MCMC sampler used to generate samples of model parameters from the posterior distribution p(c; ; ; jx; y). The second algorithm, presented in Algorithm 2, is used to summarize the posterior predictive distribution by calculating the posterior mean and variance. Details of both algorithms are shown. S is set sufficiently large to allow the Markov chain to approximately converge to the target posterior distribution. To assess convergence, we used the method introduced by Gelman et al. in [18].
Apart from mean and variance of the posterior predictive distribution, we also obtain the trustworthiness of each worker (each data point) in the form of the posterior probability of each data point being honest, or alternatively, p(cijx; y) for i = 1 n. This information can be used to further model the behavior of each worker in crowdsourcing applications. However, this is not the main focus of this paper and will not be further discussed. Derivation of posterior probability p(cijx; y) is also straightforward and thus omitted from the algorithm specification.
Assuming d n, the computational complexity of Algorithm 1 is limited by the HMC step to sample , in which L leapfrog steps are taken, each requires an inverse operation that has O(n3) complexity. This results in Algorithm 1 having an overall complexity of O(SLn3). The computational complexity of Algorithm 2 is O(S0(n2 + n n2)), where n is the length of x .
Input: S samples of posterior distribution (c(
To demonstrate that the algorithms are able to make accurate inference from data, we performed experiments on one-dimensional synthetic dataset, where n = 100 and d = 1, generated in accordance to our assumptions.
In the synthetic dataset, the underlying function is fH(x) = sin 2 x and the malicious function is fM(x) = 21 tan 4 x. 60 noisy observations are sampled from fH, with homogeneous Gaussian noise with standard deviation 0:10. The distribution of independent variables is uniform in [0; 1]. 40 noisy observations are sampled from fM, with homogeneous Gaussian noise with standard deviation 0:12. The distribution of independent variables is Gaussian with mean 0:80 and standard deviation 0:10, truncated to range [0; 1].
The choices of priors in terms of their parameters are given by: f = 2:0; f = 2:0; n = 2:0; n = 1:0; w = 0:7; w2 = 0:3;
For the Metropolis-Hastings sampler, the probability of proposing a jump is pMH = 0:05. Parameters of the proposal distribution is shown below.
pc = 0:95; F2 = 1 10 4; N2 = 1 10 4; W2 = 1
For the sampler of , hyperparameters related to H and M are sampled separately, each using an HMC algorithm with L = 10 (leapfrog steps).
Finally, for this particular dataset, we set S = 2000 since the Markov chain has converged sufficiently after 2000 iterations of the MCMC sampler. Then we pass the second half of the resulting samples to make posterior predictive inference. The results are shown in Fig. 1. Using the posterior mean as the estimator results in a Mean Square Error (MSE) of 0.0012. The posterior probability p(cijx; y) (trustworthiness) of each data point is also computed. In this example, the average trustworthiness of truly honest data points is 0:9906, while the average trustworthiness of truly malicious data points is 5 10 5. It is thus clear that for this set of data, the accuracy of the proposed algorithm is quite high.
As for the efficiency of the algorithm, we measure the average time cost by running the algorithm 50 times using the above dataset on a MacBook Pro equipped with Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz. The resulted average time cost is 27:43 seconds per run. In this paper, we build a Gaussian Process mixture model and design a MCMC-based algorithm to address the issue of collusion attacks in the regression setting. Honest data from observing underlying function and malicious data are modelled as two separate Gaussian Processes. Compared to [13], where malicious data are assumed to be observed from the same underlying function with extra noise, we claim that the mixture model is a more realistic and natural choice. We use synthetic dataset to show that our algorithm is able to produce accurate posterior predictive inference and is computationally efficient.
As part of an ongoing study, the results of this paper suggest several directions for future studies. Firstly, a single malicious function (class) is too restrictive to handle more sophisticated real-world scenarios and the algorithm currently lacks the ability to handle cases with multiple malicious groups. It is therefore reasonable to extend this model to incorporate multiple malicious classes. One way of modelling is to build a finite mixture of Gaussian Processes model with unknown number of components, and design a MCMC sampler using reversible jump techniques [20]. Another choice is to build an infinite mixture of Gaussian Processes model similar to [21]. Secondly, this model is unable to handle heteroscedasticity as mentioned before, as well as nonGaussian noises. We will explore approaches to handle these situations in future studies. Lastly, our model is not build specifically to model trustworthiness of workers, which is a significant issue in crowdsourcing applications. In our future work we will improve the model to make robust, collusion-resistant predictions while maintaining trustworthiness of workers for future predictions.