<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Markov model of evaluation of learning results</article-title>
      </title-group>
      <contrib-group>
        <aff id="aff0">
          <label>0</label>
          <institution>V.I. Serbin Astrakhan State University</institution>
          ,
          <addr-line>414056, Astrakhan, Tatishcheva, 20A, e-mail</addr-line>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2018</year>
      </pub-date>
      <abstract>
        <p>The e ectiveness of the management of the learning process depends on the adequacy and objectivity of evaluations of the learning outcomes, for which it is necessary to determine latent learning parameters, such as the training level of the trainer and the di culty of the training and testing tasks. However, the existing meth-ods of processing test results based on the Rasch model allow us to determine not the latent parameters of training themselves, but only the relationships between them. In connection with this, it is proposed to use the Markov model of training, which in-cludes three states: the state of knowledge transfer, the state of the training, and the state of knowledge control using testing. The author shows that application of the time spent by trainees to solve tasks in the framework of this model provides addi-tional information on the knowledge, abilities, and skills of trainees, assess the values of latent parameters and, based on this, better manage the learning process. The paper presents a mathematical apparatus used within the framework of the proposed model, including formalized methods of information theory.</p>
      </abstract>
      <kwd-group>
        <kwd>training system</kwd>
        <kwd>knowledge</kwd>
        <kwd>abilities</kwd>
        <kwd>skills</kwd>
        <kwd>latent parameter</kwd>
        <kwd>training</kwd>
        <kwd>testing</kwd>
        <kwd>mathematical model</kwd>
        <kwd>Markov model</kwd>
        <kwd>di erential entropy</kwd>
        <kwd>Rasch model</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>Introduction</title>
      <p>The goal of any education system is the management of the learning process. To solve this problem, it is necessary
to create a mathematical model that can ade-quately describe and analyze, and then control the learning process
[1]. The result of the operation of the education system should be an increase in the level of knowledge, abilities
and skills of the trainee. One of the important management problems in modern education system is the control
and evaluation of acquired knowledge, abilities and skills. The categories of knowledge, abilities and skills in an
automated teaching system in pedagogical work are usually determined in the following way.</p>
      <p>Knowledge is the elements of information, in which learning is understood as facts, concepts, rules, algorithms,
heuristics, the basic laws of the subject area, as well as decision-making strategies in this area, allowing people to
solve speci c production, scienti c and other tasks. However, knowledge does not mean that they will be used.</p>
      <p>Abilities are usually understood as the abilities necessary to perform a function in the work. A necessary, but
not su cient condition for the manifestation of abilities is the possession of appropriate knowledge.</p>
      <p>Skills are the ability to quickly and easily perform the actions necessary during the work. Speaking about the
skill, it is necessary to indicate a certain normative level, the achievement of which is necessary for the e ective
performance of the work.</p>
      <p>
        A rather detailed review of these methods is given in [
        <xref ref-type="bibr" rid="ref2">2, 3</xref>
        ]. Their analysis shows that the control of training
is carried out through testing, and its results are estimated in dichotomous, and at best, in ordinal scales. In this
case, the time spent on testing is either not taken into account at all, or is used only as a threshold value - to stop
the testing process. Examples of such multimedia training systems are described in [
        <xref ref-type="bibr" rid="ref3">4</xref>
        ]. The Rasch model, used
to process test results and obtain estimates, allows us to nd only the relationship between independent latent
learning parameters: the complexity of tasks and the level of training of trainees - but these values themselves
remain unknown.
      </p>
      <p>The main goal of this work is the development of a method for determining latent training parameters. A study
has been made of the connection between the time-based learning model and the Rasch model, and describes
the algorithm for managing the learning process based on the results of training and testing, taking into account
the time spent.</p>
      <p>Consider a mathematical model of learning that allows you to manage the work of the learning system in
di erent modes. This model describes the behavior of the components of the learning system as a function of
time.</p>
      <p>The object of management in this model is the learning process. The purpose of management is to improve
the quality of education.</p>
      <p>To solve the problem of managing the learning process, we present the model of the learning system as a
system with several discrete states and continuous time. We describe this model as a random Markov process
consisting of the following states: the state of knowledge transfer, the state of the solution of training tasks, or
training, and the state of control of knowledge, or testing.</p>
      <p>Figure 1 shows the Markov model of a training system of three states.
The Markov model of a learning system consists of three states:
E0 -{ state of knowledge transfer
E1 -{ state of training
E2 -{ state of testing
0 -{ transition intensity from E0 to E1
0 -{ transition intensity from E1 to E0
1 -{ transition intensity from E1 to E2
1 | transition intensity from E2 to E1</p>
      <p>Let the probability that at time t the system is in the state Ei is equal to Pi(t); i = 0; 1; 2. Then the system
of Kolmogorov equations describing the behavior of the system in time will have the form
8 dP0(t) =
&gt;
&lt; dPd1t(t) =
&gt; dPd2t(t) =
: dt</p>
      <p>0 and probabilities must satisfy the normalizing condition P0(t) + P1(t) + P2(t) = 1.</p>
      <p>This system is a Markov process of birth and death with three states with continu-ous time and constant
intensities.</p>
      <p>The paper considers three models of training:
1. The model of learning from one state, which allows you to evaluate the solution to the learner's task or
series of tasks in the state of training or testing;
2. The model of learning from two states: active learning (transfer of knowledge and training) and knowledge
control;
3. Model of learning from three states: knowledge transfer, training and knowledge control.
2</p>
    </sec>
    <sec id="sec-2">
      <title>The model of learning from one state</title>
      <p>
        and the estimate for solving the problem for a time not less than t is [
        <xref ref-type="bibr" rid="ref4">5</xref>
        ]
F (t) = 1
e
      </p>
      <p>t
p (t) = 1</p>
      <p>F (t) = e
t</p>
      <sec id="sec-2-1">
        <title>Then</title>
        <p>When t = 0 this estimate is equal 1, the rate of change of the estimate decreases with time, the evaluation
curve itself is exponential and monotonically decreasing, and t ! 1 tends asymptotically to 0, which indicates
a slow-asymptotic nature of the process, and the random variable T obeys the exponential law.</p>
        <p>
          We represent the process of solving a problem as an information processing pro-cess or as a sequence of
elementary Data Transformation Operations (DTO) [
          <xref ref-type="bibr" rid="ref5">6</xref>
          ]. Then is the number of DTO performed per unit time,
or the speed of processing information. The magnitude, the inverse of the intensity = 1= is equal to the
mathematical expectation of a random variable T - this is the average time of exe-cution of one DTO. We shall
treat this quantity as the di culty of the problem [
          <xref ref-type="bibr" rid="ref6">7</xref>
          ].
        </p>
        <p>
          The distribution density of a random variable T is
and the di erential entropy, or measure of the average information processed in the process of solving the problem,
is [
          <xref ref-type="bibr" rid="ref7">8</xref>
          ]
f (t) =
e
        </p>
        <p>t
H(T ) = ln(e= ) = ln e =
= e</p>
        <p>
          If minutes are selected as the unit of time, then is measured in min, in - dto=min where dto is one DTO,
and { in logits. Values and corresponding to it are continuous latent variables [
          <xref ref-type="bibr" rid="ref8">9</xref>
          ] and for their nding it
is necessary to use methods of statistical estimation.
        </p>
        <p>
          Let the problem be decided by n trainees. In [
          <xref ref-type="bibr" rid="ref6">7</xref>
          ] it was shown by the maximum likelihood method that the
point estimate of the di culty of the problem is equal to the mean time of solving a series of problems
n
= X ti
i=1
n = t
        </p>
        <p>The disadvantage of this model is that it takes into account only the measure of the di culty of the task and
does not take into account the level of training of the trainee.
3</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>The model of a training system that includes two states</title>
      <p>In this model it is assumed that 1 = 1 = 0</p>
      <p>
        This model describes the learning system as a random Markov process of the "birth-death" type, consisting
of two states: the state of active learning (knowledge transfer and training) and the state of knowledge control
[
        <xref ref-type="bibr" rid="ref9">10, 17</xref>
        ]. Let the random variable T be the learning time; P0(t) -{ the probability of successful completion of the
training process not later than in time t; P1(t) = 1 P0(t) -{ the probability of a successful completion of the
learning process not earlier than in time t, or the evaluation of training. The behavior of such a learning model
is described by a system of Kolmogorov equations [
        <xref ref-type="bibr" rid="ref9">10</xref>
        ]
where 0 is the intensity of the state of active learning; 0 -{ the intensity of the state of the solution of the
problem and the control of knowledge. The normalization condition P0(t) + P1(t) = 1 for any t 0 .
      </p>
      <p>Solving the system (1) for P0(0) = 0 and P1(0) = 1, we obtain</p>
      <sec id="sec-3-1">
        <title>Values</title>
        <p>P0(1) = lim P0(t) = 0=( 0 + 0); P1(1) = lim P1(t) = 0=( 0 + 0) (3)
t!1 t!1
limit values of process parameters.</p>
        <p>
          Imagine the learning process as a process of information processing or as a sequence of elementary DTO [
          <xref ref-type="bibr" rid="ref5">6</xref>
          ].
Then the intensity 0 is the number of DTO per-formed per unit time in the state of active learning. The inverse
of the intensity 0 = 1= 0 -{ the time of execution of one DTO, will be taken as the level of the student's
preparation. Intensity 0 is the number of DTO performed per unit of time in the state of knowledge control,
or the processing speed of information. The inverse of the intensity 0 = 1= 0 is the average time of execution
of one DTO. We shall treat this quantity as the di culty of the problem [
          <xref ref-type="bibr" rid="ref9">10</xref>
          ].
        </p>
        <p>( dP0(t) =
dPd1t(t) =
dt
is the probability of completing the active learning process in a time not less than t, and
is the probability of completing the process of monitoring knowledge in a time not less than t.</p>
        <p>
          From (4) follows the independence of the processes of active learning and knowledge control. According to the
local independence axiom of Lazarsfeld [
          <xref ref-type="bibr" rid="ref8">9</xref>
          ], we will assume that the quantities of 0 and 0 are also independent.
The active learning time distribution function is
and the density of distribution {
        </p>
        <p>F 0 (t) = 1
p 0 (t) = 1
e</p>
        <p>0t ;
f 0 (t) =
0e
0t:
Then the di erential entropy, or the average amount of information, is
Accordingly, the distribution function of knowledge control time is
If 0 and 0 are measured in min, then 0 and 0 in dto=min, and and in logits.</p>
        <p>As a result, the limit value of the probability of successful solution of tasks can be represented by the formula
P0(1) =
0=( 0 + 0) = 1=(1 + 0= 0) = 1=(1 + 0= 0) = 1=(1 + e
)
of i(k) and
(k) are found using iterative formulas
j</p>
        <p>
          Values 0 , 0 , 0 , 0 , and are continuous latent variables [
          <xref ref-type="bibr" rid="ref10">11</xref>
          ], and for their nding, it is permissible
that n trainers take part in the training, and each trainee solves m problems. Based on the results of solving
these problems, we make up a time table of nm size. The element of the table tij is the time spent by the
i-th student to solve the j-th problem. Let i - the intensity of the active learning of the i-th trainee, j - the
intensity of the solution of the j-th problem. In [
          <xref ref-type="bibr" rid="ref11">12</xref>
          ] it was shown by the method of maximum likelihood that
point estimates of the quantities i and j are equal to the limiting values i(k) and (jk) by k ! 1 . The values
8
&gt;
&gt;
&lt;
&gt;
&gt;
:
(k+1) = Pm
i j=1
(k+1) = Pn
j i=1
        </p>
        <p>(k)
i(k) +i (jk)</p>
        <p>(k)
i(k) +i (jk)</p>
        <p>Pm</p>
        <p>j=1 tij ; i = 1; 2; :::; n
Pn
i=1 tij ; j = 1; 2; :::; m
(5)
(6)</p>
      </sec>
    </sec>
    <sec id="sec-4">
      <title>Connection with the Rash model</title>
      <p>
        In the one-parameter model of Rasch's training [
        <xref ref-type="bibr" rid="ref12">13</xref>
        ], the equation determining the probability that the trainee
with the level of preparation s will perform the task of di culty d , the so-called success function,
P (d; s) = 1=(1 + d=s)
where d = e (or = ln d), s = e (or = ln s).
      </p>
      <p>
        Up to notation fomula (7) coincides with the formula for P0(1) , and [
        <xref ref-type="bibr" rid="ref2">3</xref>
        ] formula (8) { with the formula (5).
Thus, the education system model from two states can be considered as the development of the Rasch model,
and the Rasch success function should be considered equal to the limit value of the probability of successful
solution of tasks.
5
      </p>
    </sec>
    <sec id="sec-5">
      <title>Comparative analysis of models</title>
      <p>Let the training be conducted in three stages, and the education system can be in one of three states: the
transfer of knowledge, the development of abilities through training and testing skills through testing. Abilities
suggest that the learner simply knows how to solve the problem, and skills { that the problem should be solved
as quickly as possible. In this case, the maximum permissible or desirable time of solving the problems can be
speci ed, in their totality and/or for each task.</p>
      <p>To test knowledge, a set of control questions on the topic of my study is used. The development and testing
of skills is carried out in the course of training under the guidance of the teacher with the obligatory solution of
all training tasks. Testing skills in the testing process presupposes an independent solution for the learner of a
set of test tasks. They should be arranged in order of increasing di culty and, in addition, be adequate to the
levels of training of trainees. In the case of training, the complexity of tasks should not be lower than the level
of training of trainees, because as a result, this level should be increased. The complexity of test tasks cannot
be higher than the level of training of trainees, since these tasks include tasks that the learner must already be
able to solve, and when checking skills, the speed of problem solving is important. The choice of the relationship
between the level of complexity of tasks and the level of training of trainees at the stages of training and testing
is an independent task. These relationships can be selected, for example, by one of the iterative methods.</p>
      <p>Let n students participate in the learning process. After each stage of the training, you should check the result
with the help of test tasks. Let these tasks consist of m questions or tasks. Based on the results of the tests, we
will compile two tables consisting of n rows and m columns: table A and table T . Table A contains the results
by the dichotomous principle: aij = 1 if the i-th trainee correctly performed the j-th job and, 0 otherwise. Table
T contains a set tij { i.e. times spent by the i-th trainee for the execution of the j-th task.</p>
      <p>Let us consider the peculiarities of processing the obtained results in the Rasch model and in the model of
the training system from two states.</p>
      <p>
        The Rasch model for determining latent parameters is one of the sections of the theory of latent analysis
of Lazarsfeld [
        <xref ref-type="bibr" rid="ref13">14</xref>
        ], whose goal was to process the results of sociological surveys. Based on the results of such
surveys, tables were constructed based on the dichotomous principle. Therefore, only the table A is used in the
Rasch model, which is called the response matrix. Before processing, rows and columns containing all zeros and
all units are deleted from it. After this, the table is processed - this uses the logistic one-parameter model of
Rasch in the modi cation of Yu.M. Neumann [
        <xref ref-type="bibr" rid="ref14">15</xref>
        ].
      </p>
      <p>Thus, the Rasch model is not suitable for evaluating the results of the training, in the course of which the
learner solves all problems, and the results table A consists of one units.</p>
      <p>In the case of testing, in which skills are assessed, the time of solving the problem becomes a decisive factor in
obtaining an assessment, because skills require not only and not so much the ability to correctly solve problems
(as re ected in table A), but also solve them in an acceptable time (as re ected in table T ). Therefore, you can
expect table A of the test results to contain many rows and columns of only one unit. This makes the Rasch
model less suitable for evaluating the test results than our model of the two-state training system.
6</p>
    </sec>
    <sec id="sec-6">
      <title>Managing the learning process</title>
      <p>Based on the results obtained, the following procedure for operating the training system is proposed.</p>
      <p>
        Before starting the training with the help of an expert group, tasks are selected, problems solved and their
complexity determined [
        <xref ref-type="bibr" rid="ref15">16</xref>
        ]. Then, students are tested and the levels of their training are determined. After
that, the transfer of knowledge, training and testing to determine the level of knowledge based on the use of test
tasks.
      </p>
      <p>
        In carrying out the training, we will adopt the Yerkes-Dodson law as the axiom, which states that "as the
intensity of motivation increases, the quality of activity changes along a bell-shaped curve: rst it rises and then
gradually decreases" [
        <xref ref-type="bibr" rid="ref15">16</xref>
        ]. Therefore, the solution of the sequence of problems ordered in increasing di culty is
ful lled. First, the tasks are solved, the di culty of which coincides with the current level of training of trainees,
then the level of training of trainees is assessed. If it is increased, then the following sequence of tasks with a
higher di culty value is selected and the training continues. The process is terminated when, at the next stage,
the solution of tasks does not lead to an increase in the level of training of trainees. It is assumed that the
"maximum" result of training is achieved.
      </p>
      <p>After the training, testing is performed with the help of tasks, the di culty of which is equal to the current
level of training of trainees and the nal level of training of trainees is determined.</p>
      <p>The results of the research presented in this article will be used by the author in developing the automated
training system at the rate of mathematical logic and the theory of algorithms.
7</p>
    </sec>
    <sec id="sec-7">
      <title>The model of a training system that includes three states</title>
      <p>Since the work of any training system is divided, as a rule, into two stages, we present the work of the model of
training from three States in the form of two interrelated processes: rst, the process of knowledge transfer and
training, and then the process of training and testing.</p>
      <p>In the process of transferring knowledge and training, the system operates in the states E0 and E1. (see
Figure 4), it is assumed that 1 = 1 = 0 .</p>
      <p>In the process of training and testing, the system operates in the states E1 and E2. (see Figure 5), it is
assumed that 0 = 0 = 0 .</p>
      <p>Conclusions
1. Using the Rasch model, in which the evaluation of learning outcomes is carried out by processing responses
to test tasks presented in dichotomous and ordinal scales, it is impossible to nd the values of all latent
learning parameters, but only the ratio between these parameters.
2. The learning model described in the work, taking into account the time spent on the tasks, allows you to
nd more objective values of the assessments for the trainees, determine their level of preparation and the
complexity of the tasks.</p>
      <p>3. Ultimately, this allows you to more e ectively manage the learning process.
Danilova C. D. Adaptivnaya nechetkaya model otsenivaniya rezultatov avtomatizirovannogo testirovaniya s
razdeleniem zadaniy po urovnyam usvoeniya dis. kand. tekhn. nauk. { Ulan-Ude: VSHTU, 2005. { 122 S.
[17] Serbin V. I. Markovskaya model upravleniya protsessom obucheniya. XII Vserossiyskoe soveshchanie po
problemam upravleniya VSPU-2014. Moskva, 16-19 iyanya 2014g.: Trudy [Elektronnyi resurs] Moskva:
Institut Problem Upravleniya im. V.A. Trapeznikova RAN, 2014, s. 9489-9497.</p>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          <string-name>
            <surname>Novikov D. A.</surname>
          </string-name>
          <article-title>Teoriya upravleniya organizatsionnymi sistemami</article-title>
          .
          <source>Moskva: MPSI</source>
          ,
          <year>2005</year>
          . { 584 S.
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [3]
          <string-name>
            <surname>Zaytseva</surname>
            <given-names>L. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Prokofyeva</surname>
            <given-names>N. O. Modeli</given-names>
          </string-name>
          <article-title>I metody adaptivnogo kontrolya znaniy Educational Technology</article-title>
          &amp; Society. {
          <year>2004</year>
          . {
          <volume>7</volume>
          (
          <issue>4</issue>
          ). { S.
          <volume>265</volume>
          {
          <fpage>277</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [4]
          <string-name>
            <surname>Solovov</surname>
            <given-names>A. V.</given-names>
          </string-name>
          <string-name>
            <surname>Elektronnoe</surname>
          </string-name>
          obucheniye: Problematika, Didaktika, Tekhnologiya. Samara, Novaya tekhnika,
          <year>2006</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [5]
          <string-name>
            <surname>Serbin</surname>
            <given-names>V. I.</given-names>
          </string-name>
          <article-title>Metody otsenki znaniy v podsisteme treninga obuchayushchey sistemy Vestnik Astrakhanskogo Gosudarstvennogo Tekhnicheskogo Universiteta</article-title>
          ,
          <volume>1</volume>
          (
          <issue>36</issue>
          ),
          <year>2007</year>
          . { Astrakhan:
          <string-name>
            <surname>Izd-vo</surname>
            <given-names>AGTU</given-names>
          </string-name>
          ,
          <year>2007</year>
          . { S.
          <fpage>247</fpage>
          -
          <lpage>251</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [6]
          <string-name>
            <surname>Serbin</surname>
            <given-names>V. I.</given-names>
          </string-name>
          <article-title>Opredelenie parametrov informatsionnykh protsessov v avtomatizirovannoy obuchayushchey sisteme</article-title>
          .
          <source>Vestnik Astrakhanskogo Gosudarstvennogo Tekhnicheskogo Universiteta. Seriya Upravlenie, vychislitelnaya tekhnika i Informatika</source>
          .
          <article-title>-</article-title>
          <year>2012</year>
          . - T.
          <volume>1</volume>
          (
          <issue>53</issue>
          ). S.
          <volume>57</volume>
          -
          <fpage>61</fpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [7]
          <string-name>
            <surname>Serbin</surname>
            <given-names>V. I.</given-names>
          </string-name>
          <article-title>Metody raschota parametrov v avtomatizirovannoy obuchayushchey sisteme. Prikaspiyskiy Zhurnal { Upravlenie I vysokie tekhnologii</article-title>
          .
          <source>- 2012</source>
          . - iyun.- T.
          <volume>2</volume>
          (
          <issue>18</issue>
          ). - S.
          <fpage>66</fpage>
          -
          <lpage>70</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [8]
          <string-name>
            <surname>Kudryashov</surname>
            <given-names>B. D.</given-names>
          </string-name>
          <article-title>Teoriya informatsii. Kodirovanie diskretnykh istochnikov</article-title>
          .
          <source>Sankt-Peterburg. PITER</source>
          ,
          <year>2009</year>
          . { 315 S.
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [9]
          <string-name>
            <surname>Lazarsfeld</surname>
            <given-names>P.</given-names>
          </string-name>
          <article-title>Latentno-strukturnyi analiz i teoriya testov. Matematicheskie metody v sotsialnykh naukakh</article-title>
          .
          <source>Moskva: Progress</source>
          ,
          <year>1973</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [10]
          <string-name>
            <surname>Laptev</surname>
            <given-names>V. V.</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Serbin</surname>
            <given-names>V. I.</given-names>
          </string-name>
          <article-title>Uchet vremeni pri otsenivanii rezultatov avtomatizirovannogo kontrolya</article-title>
          .
          <source>Izvestiya Volgogradskogo Tekhnicheskogo Unversiteta</source>
          .
          <article-title>Seriya Aktualnye problemy upravleniya vychislitelnoy tekhniki i informatiki v tekhnicheskikh sistemakh</article-title>
          .
          <source>11</source>
          <year>2010</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [11]
          <string-name>
            <surname>Kochenkov</surname>
            <given-names>A. I. Tolstova Yu.N.</given-names>
          </string-name>
          <article-title>Idei latentno-strukturnogo analiza Lazarsfelda v sovremennoy sotsiologii</article-title>
          . Sotsiologiya: metodologiya, metogy, matematicheskie modeli. {
          <fpage>2003</fpage>
          -
          <lpage>16</lpage>
          . { S.
          <fpage>127</fpage>
          -
          <lpage>149</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [12]
          <string-name>
            <surname>Serbin</surname>
            <given-names>V. I.</given-names>
          </string-name>
          <article-title>Metod rascheta parametrov avtomatizirovannoy obuchayushchey sistevy iz dvukh sostoyaniy. V mire nauchnykh otrpytiy</article-title>
          . {
          <year>2012</year>
          . { E.
          <volume>1</volume>
          (
          <issue>25</issue>
          ). { S.
          <fpage>153</fpage>
          -
          <lpage>166</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [13]
          <string-name>
            <surname>Rasch</surname>
            <given-names>G.</given-names>
          </string-name>
          <article-title>Probabilistic Models for Some Intelligence</article-title>
          and
          <string-name>
            <given-names>Attainment</given-names>
            <surname>Tests</surname>
          </string-name>
          . Copenhagen, Denmark: Danish Institute for Educational Research,
          <year>1960</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [14]
          <string-name>
            <surname>Khlebnikov</surname>
            <given-names>V. A.</given-names>
          </string-name>
          <article-title>Teoreticheskie osnovy obyertivnogo izmereniya uchebnych dostizheniy uchashchikhsya</article-title>
          .
          <source>Moskva: Federalnoe gosudarstvennoe uchrezhdenie Federalnyi tsentr testirovaniya</source>
          ,
          <year>2005</year>
          . { 127 S.
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>Neyman</given-names>
            <surname>Yu</surname>
          </string-name>
          . M,
          <string-name>
            <surname>Khlebnikov</surname>
            <given-names>V. A. Vvedenie</given-names>
          </string-name>
          <article-title>v teoriyu modelirovaniya i parametrizatsiyu pedagogicheskogikh testov</article-title>
          .
          <source>Moskva: Prometey</source>
          ,
          <year>2000</year>
          . { 169 s.
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Kharchenko</surname>
            <given-names>M. A.</given-names>
          </string-name>
          <article-title>Korrelyatsionnyi analiz. Uchebnoye posobie dlya vuzov</article-title>
          .
          <source>Voronezh: Izdatelstvo Voronezhskogo Gosudarstvennogo Universiteta</source>
          .
          <year>2008</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>