=Paper=
{{Paper
|id=Vol-2914/Paper31.pdf
|storemode=property
|title=Modeling of Expert Estimation
|pdfUrl=https://ceur-ws.org/Vol-2914/paper31.pdf
|volume=Vol-2914
|authors=Antonina V. Ganicheva,Aleksey V. Ganichev
}}
==Modeling of Expert Estimation==
https://ceur-ws.org/Vol-2914/paper31.pdf
340
Modeling of Expert Estimation*
Antonina V. Ganicheva 1[[ 0000-0002-0224-8945] and Aleksey V.Ganichev 2[0000-0003-3389-7582]
1Tver State Agriculture Academy, 7 Marshala Vasilevskogo Street (Sakharovo),
Tver, 170026, Russia
2Tver State Technical University, 22 Af. Nikitina Embankment, Tver, 170026, Russia,
alexej.ganichev@yandex.ru
Abstract. The problem of collective decision-making by a group of experts is a crucial
one in the theory and practice of decision-making. To obtain a high-quality collective
estimation of an object or process, individual expert opinions should be coordinated. In
this article, we use the absolute error of estimation (deviation of individual expert
estimates from their arithmetic mean) and the relative error (absolute error divided by the
arithmetic mean of estimates) as indicators of individual estimates consistency within a
collective decision. We consider errors as random variables that fall under the normal
probability distribution law. For the selected indicators, their variances, probability
distribution densities, and confidence intervals (for averages and variances) are obtained.
The preference criterion of methods for constructing confidence intervals is obtained for
variances based on the number of experiments. An algorithm is given for finding the
required number of experiments when the variance of estimation errors is not specified.
A method is developed for obtaining the distribution density of the relative estimation
error, its average value, the spread, and the probability of falling within a certain range
of values, with a given degree of accuracy. We consider the determination of the required
number of experiments (the number of questions or test tasks) for obtaining reasonable
estimates by students based on the test results, as the practical implementation of the
proposed methods of expert estimation.
Keywords: Sampling, Estimation Error, Confidence Interval.
1 Introduction
The expert estimation method is used to solve difficult to formalize problems for which
classical optimization methods cannot be applied. To choose the best solution, in this
case, the methods of voting (simple, weighted) and detachment of committees and co-
alitions are used. In the case of collective estimation, the task of coordinating the indi-
vidual opinions of experts emerges. For random estimates, the minimum sum of the
coefficients of variation or the minimum sum of centered estimates (we call it the esti-
mation error) may be used as an indicator of decision consistency. Not only specialists
in a certain field of knowledge may be considered experts, but testing systems, intelli-
gent agents, algorithms, diagnostic systems, measuring instruments, and neural net-
works as well. The use of expert estimates in the case of distance learning is of special
* Copyright 2021 for this paper by its authors. Use permitted under Creative Commons License
Attribution 4.0 International (CC BY 4.0).
341
interest. Testing of each trainee may be considered a collective estimation of their
knowledge concerning different domains (topics, sections, lectures, practical training,
and laboratory classes) of a certain discipline. In this case, the collective body of experts
represents the questions and tasks of a test that are considered experiments.
The reference literature pays much attention to the methods for determining the op-
timal combination of the confidence interval and the confidence probability [4, 12], the
use of Chebyshev inequality for making the confidence intervals of expert estimates
[5], making the robust (stable and independent of the type of distribution law) estimates
[9], and the determination of the degree of various factors influence upon the result
based on expert estimation [7]. A recent trend is the use of artificial intelligence meth-
ods in expert estimation: the methods of subjective probabilities [8] and fuzzy intervals
[10]. The expert estimation models are used to determine the quality of the education
process [11].
The objective of the article is to develop mathematical models for consistent expert
estimates and estimates of collective decision quality.
2 Setting the Task of Expert Estimation
Let us assume that n independent experts estimate some object A. Let xi i i, n
1 n
denote the estimate given by the ith expert and let x xi denote the average esti-
n i 1
mate given by all experts. Data on the expertise conducted by those n experts may be
considered as the results of n experiments. All possible estimates of object A form a
general population of values for some random variable X.
Before experimenting, we will consider the sample units X1 , X 2 ,..., X n to be pair-
wise independent random variables that have the same distribution law as X has. Let us
assume that X has a normal distribution, mx and x is its mathematical expectation and
mean square deviation, accordingly. The variable x is random, since it is determined
by the sample [1], has a normal distribution (as the sum of normally distributed varia-
x2
bles), and also M x mx , D x .
n
Let us assume that x is an arbitrary value of a random variable X. Then the variable
y x x characterizes the degree of consistency of the experts' opinions. This variable
will be called an absolute error of estimation.
We assume that the interval xmin , xmax (here, xmin and xmax are correspondingly
the minimum and maximum values of the random variable X) coincides with the inter-
val u1 , u2 mx 3 x , mx 3 x and mx 3 x 0 . If the first interval is less than the
second one, we stretch it by the appropriate number of times, moving to the new xmin
and xmax .
Now the distribution density of the random variable Y may be found.
342
3 The absolute error of estimation
Consider a random variable Y X x . Using the composition of two normally distrib-
uted random variables, we may obtain the distribution law Y. When subtracting random
variables, their mathematical expectations are subtracted, and the variance is calculated
using the formula:
n 1
Dx , если x x1 ,..., xn ,
n
D Y D X x D X D x 2 K X , x
n 1 D , если x x ,..., x .
n
x 1 n
The distribution density of the random variable Y is equal to:
x x n
2
n
2 n 1 e
2 n 1 x2
, если x x1 ,..., xn
f y f x x
x
x x n
2
. (1)
n
e 2( n 1) x , если x x1 ,..., xn
2
2 n 1 x
Note that n n 1 with an accuracy of 3.3 % if n n0 40 .
In this formula, m y 0 and 99.73 % of Y values fall within the interval u1 , u2 .
It is inconvenient to make calculations using the formula (1) because x2 is un-
2
known. Consider replacing x2 with S x2
1 n
xi x . With such a replacement,
n 1 i 1
there will be a calculation error.
Let us find the number n, at which the indicated replacement will take place with
the specified accuracy and probability . The problem is reduced to constructing a
confidence interval for the variance.
Without violating generality, we compare two methods (described in [1] and [5])
for constructing a confidence interval for the variance.
[1] discusses the approximate method for constructing a confidence interval for the
variance x2 when the number of observations n 30 . It is the interval
2
S x t
2
n 1
S x2 , S x2 t
2
n 1
S x2 , where is the accuracy of
the estimate and t is the argument of the Laplace function for the confidence proba-
bility β. This suggests
2t2 S x2
2
n 1. (2)
2
[3] discusses the method for constructing a confidence interval for the variance when
n 30 . The confidence interval looks as follows:
343
n 1 S x2 n 1 S x2 .
,
n 1 t 2(n 1) n 1 t 2(n 1)
Then
2 n 1 S x2 t 2(n 1) 2 S x2 t 2(n 1)
S x2 x2 .
n 1 2t2 (n 1) n 1 2t2
2
In this case,
8t2 S x4
n 4 t2 0,52 t4 1. (3)
2
We obtain a sufficient condition for the second method to prevail over the first one
relative to the number of experts.
Let us assume that 1 S x2 , i.e. 1 is the fraction of in S x2 . Then (2) and (3)
will correspondingly look as follows:
2t2 4t2 1 2t2 12
n 1,
12
8t2 4t2 12 0,52 t4 12
n 1.
12
Now we find the difference between the right-hand members of the latter inequalities
(subtracting the first one from the second one). The result is:
6t2 0,52 t4 12 2t2 12
.
2
1
Hence it is not difficult to demonstrate that 0 if and only if the following inequality
is correct:
1112 41 6 0.
Maximum value t2 25 . Then 0 if and only if
1 0,579.
This means that Sx2 x2 0,775Sx2 .
Thus, if Sx2 x2 0,579Sx2 then the number of experts, according to the second
method, will be less than those under the first method.
This is a criterion for the second method prevailing over the first one in terms of the
number of experts.
We will use the first method.
We assume that 2 S x2 2 S x2 and 0 2 2 1 , then the condition (2) is con-
verted into the following:
2t 2 (1 ) 2
n n1 1.
2
(4)
2
2
344
So, for 0,1; 2 0,15; 2 0,09 we find: n 292 .
When calculating the relative error of x2 replacement by S x2 we have:
x2 S x2 1 2 S x2
2 .
2
x S 1
2
x S 2 S x2 1 2
2
x
Thus,
x2 S x2
2
. (5)
1 2 2
x
Inequality (5) is good under n n1 probability 1 .
Here is an algorithm for finding n when S x is not specified; iterations are made to
determine the value S x so that the inequality (2) is valid. Let us assume that
n2 max n0 , n1 .
Steps of the algorithm (we will call it Algorithm 1).
1. Given: β, ε, n2 , nmax is the maximum possible number of experts in this situation.
2
2. Find the right member of the inequality (2) when S x2
1 n
xi x
n 1 i 1
and
1 n
x xi . For this purpose, estimates of n2 independent experts are considered.
n i 1
3. If the right member of (2) obtained in step 2 is not less than n2 and not more than
nmax , we get an estimate for the number of experts n so that the inequality: n2 n nmax
is valid. Go to the End.
4. If the right member of (2) is less than n2, then we assume n2 n2 1. At that, if
n2 nmax , move to step 2.
5. Otherwise, it is necessary to decrease the value or increase the value β.
If n n2 max n0 , n1 the formula (1) is converted to the formula
x x
2
1
f1 y f1 x x e 2 S x2
, (6)
2 S x
since for these values of n x2 S x2 with probability 1 and with accuracy .
It may be demonstrated that the relative error of formula (5) does not exceed
1 ( x x) 2
3 1 2 . (7)
2 x x n 1 1 2
with probability .
4 The Relative Error of Estimation
When estimating the errors made by experts, an important role is assigned to the relative
345
xx xx
error of estimation; such an error may be defined either as or by 1
x x
xx
(the case is symmetric to 1 ).
x
xx xx
Let us assume that is a relative error. If 1 , then the estimate x
x x
xx
is not consistent with x ; if 1 , then x is consistent with x and the better the
x
smaller the relative error is.
If is no more than 3 %, then the estimate x has increased accuracy; if falls
within the range from 3 % to 10 %, then this is the usual accuracy; from 10 % to
20 % results in an approximate estimate [2].
xх
Consider 1 as a relative error. This is the value of a random variable ∆1,
х
1 n
represented by the ratio of variables Y X х and х хi , which have a normal
n i 1
distribution, moreover, M Y 0 , M х mx , if n n2 with probability 1 and
S x2
with accuracy D х
.
n n
One may replace an unknown value mx with an exact estimate x that meets the
condition:
S S
x x t , n 1 mx x x t , n 1 , (8)
n n
where t ,n1 is the argument of the Student function t , k , which is such that
t ,n 1 , n 1 1 and
S x2 t2 , n2 1
n n3 max n2 , . (9)
42
The relative error of estimate mx :
x mx
6 4 4 ,
mx mx x 4
where 4 x 5 and 0 5 1 , i.e.
x5
6 5 . (10)
x x 5 1 5
[6] proved that has a normal distribution if n n3 and 2 S x2 max 2 S x2 , x 5 ,
346
while
1 2
1
f 2 1 e 2 S x2 с 2
, (11)
2 S x с
where
1 1 S x2
c . (12)
mx m 3 n
x
Then
d
P d 1 d Ф . (13)
Sx c
So, formulas (11) to (13) give a fairly accurate value of the distribution density of the
relative estimation error, its average value, the spread, and the probability of falling
within the range d , d .
Here is the algorithm for solving these problems.
1. First, Algorithm 1 is applied. If it ends with Go to the End, move to step 2.
2. n3 is calculated.
3. If n2 is equal to n3, then n n2 . Move to 4.
Otherwise, we assume n2 n2 1 and move to step 2 of Algorithm 1.
4. Calculation of the spread of the relative estimation error.
4.1. First, we calculate “c” using the formula (12).
4.2. Use the right member of (12) for n n1 .
5. Calculation of probability of the relative error falling within the interval d , d
where d is specified.
5.1. “c” is calculated using the formula (12).
d
5.2. The value of the Laplace function at a point is calculated.
Sxc
5 Practical Implementation
When we tested students in the probability theory and mathematical statistics at the
Tver State Agriculture Academy, the maximum score was 10 points, taking into ac-
count the complexity of the test. We used prompts (no more than 5); each of the prompts
reduces the score by 0.7 k points, where k is the number of useful prompts. The maxi-
mum number of prompts, which is equal to 5, was determined based on these trial tests
that allowed consulting. The decrease in the score occurs in arithmetic progression. The
parameter 0.7 is selected from the condition of the maximum “penalty” for a prompt
because in this case the score is reduced for five prompts to the maximum level. If the
answer to this task is incorrect, two approaches are possible. The first (standard) ap-
proach: a score of 0 points is graded for the task regardless of the number of prompts
347
the student used. The second approach is to use estimates of ontologies, various frag-
ments of this task, and methods of fuzzy control in an adequate system for estimating
the quality of teaching.
When selecting the volume of n=300 test tasks, on average, the group of students
got: x 5,1; S x2 3, 06 and 𝑛 ≥ 292 if 0, 27 .
When statistical data were approximated by the normal distribution law, the signif-
icance level was 0.1.
Note that the results of this article are valid for traditional testing without prompts
as well. At the same time, well-defined prompts greatly contribute to the use of the test
not only for monitoring but also for training, i.e. they increase the teaching potential of
test tasks.
6 Conclusion
Creating reliable and high-quality methods of making collective decisions by experts is
a significant top-of-the-agenda topic of modern research in the field of complex systems
modeling. This issue is crucial for remote testing of learners.
The developed method enables to obtain quantitative estimates of the required num-
ber of experts to make a joint decision of a given quality.
This method may be used not only for testing learners, but also in diagnostic sys-
tems, product quality control, and other areas as well.
References
1. Ventcel', E.S.: Probability Theory: Textbook. JUSTICE (2018).
2. Ganicheva, A.V.: Mathematical Models and Methods for Evaluating Events, Situations, and
Processes: Textbook. Saint Petersburg: «Lan'» (2017).
3. Mihin, M.N.: Mathematical Statistics: Textbook. Moscow: MIREA (2016). (2016).
4. Simankov, V.S., Buchackaya, V.V., Teplouhov, S.V.: Determination of the Optimal Com-
bination of the Confidence Interval and Confidence Probability. The Bulletin of the Adyghe
State University. Series 4: Natural-Mathematical and Technical Sciences. 3 (246), 69-74
(2019). (In Russian)
5. Suvorova, A.V., Pashchenko, A.E., Tulup'eva, T.V., Tulup'ev, A.L.: Construction of Confi-
dence Intervals for Assessing the Intensity of Risky Behavior Based on Chebyshev's Ine-
quality. SPIIRAS Proceedings. 10, 96-109 (2009). DOI: 10.15622/sp.10.6.
6. Ganicheva, A.V.: Test Technologies in Training. Tver: TGSHA (2011).
7. Halikova, K.S., Ryzhkova S.K.: Assessment of the Influence of Factors Based on Cognitive
Modeling and Expert Assessment. Humanities Research. 2 (54), 300-303. (2016).
8. Boiko, Ye.: Methods of Forming an Expert Assessment of the Criteria of an Information
System for Managing Projects and Programs Technology Transfer. Fundamental Principles
and Innovative Technical Solutions. 2, 9-11 (2018). DOI: http://dx.doi.org/10.21303/2585-
6847.2018.00766.
9. Bonett, D.G.: Robust Confidence Interval for a Ratio of Standard Deviations. Applied Psy-
chological Measurement. 30(5), 432-439 (2006). DOI: 10.1177/0146621605279551.
348
10. Chakraborty, D.: Structural Quantization of Vagueness in Linguistic Expert Opinions in an
Evaluation Program. Fuzzy Sets and Systems. 119 (1), 171-186 (2001)
DOI: 10.3745/KIPSTB.2005.12B.2.203.
11. Safargaliev, E.R., Eremina, I.I., Savitsky, S.K., Camelina, V.A.: Mathematical Model and
Qualimetric Assessment of Graduate Education Quality in Environment Saturated with In-
formation and Communication Technologies. International Education Studies. 8(2), 78-83
(2015). DOI: 10.5539/ies.v8n2p78.
12. Thulin, M.: On Split Sample and Randomized Confidence Intervals for Binomial Propor-
tions. Statistics & Probability Letters. 92, 65-71 (2014). DOI: 10.1016/j.spl.2014.05.005.