There are some situations when we need to compare work results of dierent people. For example, it may be an announced tender, a grants' distribution or a competition in poetry. And sometimes it is allowed for one participant to present more than one work. It is extremely actual when we want not only to identify the winner but also to get a lot of eective projects. We don't want to bound the author's creativity therefore. Then having expert evaluations we must obtain integral evaluation for each work. How to do this? Also it may be useful to evaluate author's level on the basis of his works' evaluations. Moreover, we may want to evaluate expert's level on the basis of his evaluations. In this paper we solve these problems by constructing a Bayesian network model for such a competition and applying a maximum a posteriori estimation method to it.
Vorobev Alexandr
Let us take a competition between projects, which allows an author to present more than one project. The participants present their projects and experts evaluate these projects, not more than one evaluation given by one expert for one project.
The aim is to estimate level of each project, level of each expert and level of each author on the basis of expert evaluations.
We introduce the following model of the competition.
1) Each author has his level C(Creator) - degree of his ability to create the eective projects.
2) Each expert has his level E (Evaluator) - degree of his ability to give for a project the proper evaluation. 3) Each project has its level L(Level) - degree of its eectiveness. This level is random value and appears when its author creates this project. We also assume that L has a distribution depended on its author level C - RandL(C).
4) Each project evaluation e given by an expert is also a random value and its distribution depends on the project level and on the expert level - Rande(L; E). 5) The number of projects presented by author is independent from his level.
Distributions RandL(C) and Rande(L; E) may be xed if appropriate parameters are xed or may have another parameters for which we don’t have prior exact values. Then in general case we have distributions RandL(C; P ) and Rande(L; E; Q) where P and Q are vectors of parameters.
Thus we can formulate the following problem: Given 3-dimensional vector of project evaluations where ei;j;k - the evaluation given by the expert i to the kth project of the author j (we number projects of one author beginning from the one); mj - number of the projects of the author j.
The evaluations of the following values have to be found: (Cj )j=1::n - vector of the author levels; (Ej )j=1::n - vector of the expert levels; (Lj;k)j=1::n;k=1::mj - 2-dimensional vector of the project levels; P - vector of the distribution parameters RandL; Q - vector of the distribution parameters Rande.
Thus we have constructed a Bayesian network as a model for our competition. 2
It may be interesting to nd posterior distribution for unknown parameters in this model too, but here we are seeking for parameters’ estimates only. The standard method for this problem is the maximum a posteriori estimation method. We do not have any special suggestions about the parameter distribution. Therefore we consider the prior parameter distribution to be uniform. In this case maximum a posteriori estimates coincide with maximal likelihood estimates.
It means that the estimate of a parameters’ vector equals such a value, that conditional probability of the fact, that the expert evaluations are equal to their observed values is maximal.
Up to this moment all our values could lay in any set. But we need to specify
the kind of these sets to write the formulae. We consider the nite sets due to
the following reason. It is well known (see [
Then it is natural for evaluations ei;j;k and project levels Lj;k to lay in the same set. Then evaluations can be considered as project level evaluations.
Let us also consider the values C, E, P , Q to be discrete. It makes the results more simple to be interpreted (values C, E) and makes the narrative more simple.
To simplify our gures we drop indexes in the names of vectors if they may have all the possible values. So we write (Ai;j;k)k=1::mj instead of (Ai;j;k)i=1::n;j=1::n;k=1::mj and (Ai;j ) instead of (Ai;j )i=1::n;j=1::n.
Also we mean that the index under the operator of sum or multiplication runs through the whole set of its possible values.
Then maximum a posteriori estimates have the following form: ((Lfj ); (Cfj ); (Efj ); Pe; Qe) = = arg max (Lj;k)k=1::mj ;(Cj);(Ej);P;Q
P ((ei;j;k)k=1::mj j(Lj;k)k=1::mj ; (Cj ); (Ej ); P; Q) = = arg = arg
max Y (P (ei;j;kjEi; Q; Lj;k)P (Lj;kjCj ; P )) = (Lj;k)k=1::mj ;(Cj);(Ej);P;Q i;j;k
max X(lnP (ei;j;kjEi; Q; Lj;k) + lnP (Lj;kjCj ; P )) (Lj;k)k=1::mj ;(Cj);(Ej);P;Q i;j;k
It is a problem of discrete optimization which is NP-complete. It means that the complexity of computations of an exact answer is huge for the big number of parameters. Hence we have to use optimization methods which give an approximate answer.
There are dierent optimization algorithms for this problem, for example the expectationmaximization algorithm, the gradient method, the analogue of belief propogation algorithm. The choice of such a method depends on the model. Probably we should try dierent methods to understand which is better in our case. (1)
Here we specify the distribution functions RandL(C; P ) and Rande(L; E; Q). As we decided random values Lj;k and ei;j;k are distributed on the same nite set. This set has to be small because of the big complexity of the optimization problem solving. In our data this set is the set if integers from the interval [ 3; 3].
Then we consider the following assumptions to be natural: 1) The mathematical expectation of the expert evaluation ei;j;k equals the true level of this project Lj;k.
2) Its volatility depends on the expert level: the higher the expert level, the less the volatility. It means if we give the big set of projects of the same level to the expert (even bad) the average of his evaluations will be close to this true level. But the better the expert the less standard deviation of his evaluations will be.
It is the best practice to use normal distribution in the models due to the existence of the central limit theorem. But we have discrete sets of variables. Hence we use the value of normal density divided by normalizing constant as the probability function value for each discrete point. Then the probability function has the bell curve shape.
Now we have to dene the volatility as function of the expert level. Let us
take volatility to be equal to Eqi . We need constant q because Ei is bounded (it
lays in the nite set), but we don’t know the bound for volatility - it depends on
the data. And we suppose that Ei is integer from the interval [
Thus value ei;j;k is distributed according to the law
P (ei;j;k = ajLj;k; Ei) = 3::3
(2) e 3 P e b= 3 (a Lj;k)2Ei2 2q2 (b Lj;k)2Ei2 ; a = 2q2
We dene the same discretized normal distribution for the project level Lj;k where the mathematical expectation equals the author level Cj and the volatility equals the constant parameter p. It means the higher the author level the higher the average level of his projects. Volatility is constant because we don’t know any facts about the dependence between it and the author level.
Then Cj is the integer from the interval [
Thus value Lj;k is distributed according to the law:
P (Lj;k = ajCj ) =
There is a standard problem which can be formulated in the terms of this paper as the following: it is necessary to evaluate the project levels and the expert levels on the basis of the evaluations given by these experts to these projects.
Except our attempt to evaluate the author level there is the principal difference between our approach to this problem and standard approach. In our problem we have authors of the projects. And we think that the projects of one author will more probably be similar in their levels than the projects of two dierent authors. This fact underlies our model.
The standard approach (see [
The paper presents a new model of a competition. Within this model a maximum a posteriori estimation can be applied to estimate the project levels, the project author levels and the expert levels on the basis of the evaluations given by these experts to these projects.
Notice that the presented algorithm does not have any requirements for the existence of the evaluation given by the denite expert to the denite project or for the number of evaluations per one project. Another question is that the more evaluations we have the more exact the results of the algorithm are.
We are going to apply this method to the data in the nearest future. Then we will be able to compare this algorithm with the standard approach in estimation of ((Lj;k)k=1::mj ; (Ej )).
Also we are going to consider this model when the authors are the experts at the same moment. Then in our model we can take into account a correlation between participant level as an author and his level as an expert. Or we can evaluate this correlation on the basis of the results of this method if we don’t take it into account in the model.