=Paper=
{{Paper
|id=Vol-2770/paper22
|storemode=property
|title=Qualimetric Approach to Dynamic Evaluation of Educational Activities According to Facet Classification of the English Language Tenses
|pdfUrl=https://ceur-ws.org/Vol-2770/paper22.pdf
|volume=Vol-2770
|authors=Mikhail Noskov,Petr Dyachuk,Irina Peregudova,Oleg Denisenko
}}
==Qualimetric Approach to Dynamic Evaluation of Educational Activities According to Facet Classification of the English Language Tenses==
https://ceur-ws.org/Vol-2770/paper22.pdf
Qualimetric Approach to Dynamic Evaluation of
Educational Activities According to Facet Classification
of the English Language Tenses*
Mikhail Noskov1 [0000-0001-8966-3633], Petr Dyachuk 1 [0000-0003-1555-7250],
Irina Peregudova1 [0000-0002-1361-7226], and Oleg Denisenko1 [0000-0001-5832-1040]
1
Siberian Federal University, ul. Akademika Kirenskogo, 26k1, 660074 Krasnoyarsk, Russia
mvnoskov@yandex.ru
Abstract. On the basis of the developed facet dynamic adaptive tests-
simulators, there were obtained protocols of educational actions numerical
evaluation on facet classification of English language tenses, for a random
sample of 150 students. Testees are taught the facet classification of puzzles
with sentences in English in a randomized puzzle electronic problem
environment, in the conditions of numerical reinforcements of the actions of a
testee: + 1 correct; - 1 incorrect actions. A self-consistent change in the relative
frequency of reinforcements helps to adapt and increase the level of autonomy
of testees. The qualification approach to the dynamic evaluation of the
educational activity protocols made it possible to specify three groups of
testees: those who have achieved the state of autonomous educational activity;
approaching the state of autonomous educational activities; with insufficient
training in foreign languages or with "trained" helplessness to bilingual
activities
Keywords: qualimetry, educational activity, adaptation, dynamic assessment,
procedural characteristics, actiogram, entropy, labor intensity
1 Introduction
The problem of the qualimetry of the dynamic evaluation of the procedural
characteristics of educational activities is topical and still not completely solved [1].
Dynamic assessment refers to a procedure that combines testing and training into a
single procedure aimed at simultaneous understanding and advancing the abilities of
students through indirect interaction in the zone of immediate development [2]. The
idea that the student's learning potential is determined by the zone of immediate
*
The research was carried out within the state assignment of Ministry of Science and Higher
Education of the Russian Federation (theme No. FSRZ-2020-0011). Supported by the
Krasnoyarsk Territory Science Fund, grant No. 12/19.
Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0
International (CC BY 4.0).
Proceedings of the 4th International Conference on Informatization of Education and E-learning Methodology:
Digital Technologies in Education (IEELM-DTE 2020), Krasnoyarsk, Russia, October 6-9, 2020.
development is fundamental in the sociocultural theory of human development of L.S.
Vygotsky [3].
As stated in work [4], "teaching with a teacher is teaching by examples presented
by some informed external authority." The student's feedback to the mediator or
teacher is instructive [5; 6]. It consists of instructions, recommendations, regulations,
advice, etc., which the teacher gives to the student, contributing to the student's
development. Under the conditions of instructive feedback, the dynamic evaluation
involves the interactive interaction of the testee with the intermediary or mediator [7].
Dynamic assessment allows the diagnosis of learning potential, which is determined
by the nature of the change in the academic activity.
Despite the complexity and hardness of computer simulations of mediator
activities, electronic systems for dynamic assessment of the learning process are
developed in various subject areas [8; 9]. The most successful is the attempt of
computerized dynamic assessment development of teaching a second foreign
language [10]. However, the interactive interaction of the mediator or teacher with the
student is so complex that its modeling is often reduced to a rigid scenario of the
mediator's reactions to the difficulties of the testee. The procedure of dynamic
evaluation with a rigid scenario of intermediary reactions is called the interventionist
one [11]. A dynamic evaluation procedure in which there is no rigid scenario of
intermediary reactions, and interactivity is situational in nature and is called the
interactionist one [11].
In our study, we used the ideas of reinforcement learning [6] to dynamically
evaluate the learning process of problem solving of the facet classification of the
English language tenses. Following the dictionary of Vishnyakov S.M. [12], we
define facet classification as a classification system, which basis is the objects or
concepts division in one aspect, by one characteristic.
2 Methodology
Learning of facet classification, based on ideas of reinforcement learning, is
fundamentally different from the traditional approach of learning with a teacher.
Feedback on reinforcement learning is evaluative. That is, the actions of the student
receive reinforcements in the form of a numerical assessment. The evaluative nature
of feedback determines the search activity of trainees in an electronic problem
environment. Receiving reinforcements from the problem environment for his actions,
the testee must learn to perform tasks. The complexity of the problem environment is
determined by the changing uncertainty of tasks and reinforcements. A testee adapts
to the problem environment in the course of educational activity when learning how
to solve problems based on the experience of his interaction with the problem
environment. In the process of adapting a student to a problematic environment,
autonomous educational activity is formed.
In the process of learning how to solve problems, two components of the
educational activity of the subjects can be distinguished: the first is the study of the
problem states space; the second is the application of the obtained information about
the task state space. As a result, two distinctive characteristics of the interaction of the
testee with the electronic problem environment are determined - the ability to conduct
search by trial and error and delayed incentives [14].
The learning with evaluation feedback includes the interaction of the student with
the problem environment, which contains uncertainty, undefined factors. The testee
searches for a solution to the problem, despite the presence of uncertainty in the
training environment. He makes the decision to choose the action himself. At the
same time, he can evaluate the nature of the progress towards the desired goal, based
on a direct perception of the current state of the task. The student uses the
accumulated experience to improve his characteristics over time. The knowledge that
students bring to the task at the beginning of the process of its solving has an impact
on how useful or simple the process of self-teaching will be. But the decisive factor
that ensures the adjustment of the student's behavior taking into account the specific
features of the problem being solved is his interaction with the problem environment
based on evaluative feedback.
According to the law of influence of Thorndike [15], actions leading to good
(rewards) or bad consequences will respectively be repeated or rejected. The law of
influence combines search and memorization processes. At the same time, the search
is interpreted as a test of various options for actions in relation to each of the
situations with the choice of one of these options, and memory - as a way of saving
actions that made it possible to get the best result, associating them with those
situations or states of solving the problem in which they gave this result. The
combination of search and memory in the conditions of evaluative feedback
characterizes the adaptation of the student to educational activities when learning how
to solve problems. The evaluative feedback assumes that the current problem solving
states are matched with the value of the task state equal to the total reward that the
student will receive when performing actions that transfer him from the initial to the
current problem solving state.
For self-management of educational activities, it is important for a student to have
alternative options for finding solutions to problems and multitude of actions
(operations). Since the information, according to the definition of Quastler H, " is a
random and memorable choice of one option from several possible and equal ones,"
the student should have the opportunity to independently make decisions about
choosing an option from the formed set [14]. According to the sociocultural theory of
development of the Vygotsky L.S., the zone of actual development [3] of the student
should allow him to form many options for actions and ways to find solutions to
problems. Choosing an option gives the student information of value to achieve the
goal. The area of actual development (its thesaurus) of the testee should contain
elements (information) corresponding to the level of complexity of solving problems.
In dynamic evaluation computer systems, conditions must be created so that the
testee can show creative independence in converting the original information into new
messages. The factor of semiotic diversity is a prerequisite for adaptation of the
subject to educational activities in electronic problem environments. Semiotic
diversity allows the subject to realize the potential of learning, develops the ability to
operate with numerical and sign symbols, as well as formal structures, structures of
relations and connections [17].
As noted above, learning with reinforcements in dynamic learning potential
assessment systems is based on the fact that the testee attempts to maximize the
remuneration received by operating in an electronic problem environment with a high
level of uncertainty. Uncertainty of problem environment is defined by changing
conditions of solving problems, introduction of elements of randomness of parameters
of problem environment and problems, as well as setting of many alternative options
for selection of training actions and ways of finding solution of problems. Uncertainty
causes an imbalance between cognitive needs and cognitive capabilities, initiating
bifurcation [16] of the study activity of the subject (see Figure 1).
The qualimetry complexity of the procedural characteristics of the educational
activity is due to the fact that they are usually qualitative characteristics of the ability
of the student to produce and absorb new information during the training process.
Search of a solution to the problems of classifying sentences in English by facets of
English tenses consists in recognition of mismatch between the current and target
states of solving the problem and execution of actions that reduce this mismatch to
zero.
Sentences in English are located on the puzzles, which a testee must relate to
facets. If the action is performed correctly, then a testee will receive positive
reinforcement in the form of a numerical estimate of + 1. If the action is incorrect,
then the testee receives reinforcements in the form of a numerical estimate - 1.
The qualimetric approach in the dynamic evaluation of the sentences classification
process in English by English language tenses in facet dynamic adaptive tests-
simulators [19] is implemented by:
- tracking and recording of educational activities of the trainee in real time;
- recognition of the mismatch value between the current and target status of the
task solution and its correction through local feedback mechanisms in the form of
information about the "distance to the target" in on-line mode;
- self-consistent adjustment of relative frequency of learning actions
reinforcements during the current task performance depending on relative frequency
of correct actions during performance of the previous task.
The procedural characteristics diagnostics qualimetry is based on a numerical
assessment of the educational actions of a testee, which play the role of
reinforcements. Computer tracking of target task state search is performed in on-line
mode. For example, the sequence of single rewards and fines is: 1; 1; 1; 1; -1; -1; 1; -
1; 1; 1; 1; 1; -1; -1; -1; 1; 1. The total win is + 5. The total gain is equal to reducing
the distance (in actions) to the target. Sequences of educational actions and
reinforcements form time series of events recorded in the protocol. In zero
approximation, educational actions have three numerical estimates of 1, 0. -1. The
correct action reduces the mismatch between the current state and the target state, and
has a numerical estimate - reward + 1.
Fig. 1. Activity protocol fragment of a test
An educational activity that does not explicitly change the mismatch between the
current and target status of the task (e.g., browsing, listening, etc.) has a numerical
value of 0. The wrong action has a numerical rating - fine - 1. It increases the
mismatch between the current and target states.
The time series of events - educational actions presented in the protocols (see
Figure 1) form the qualimetric basis for the procedural characteristics of training
activities. Figure 2 shows a graph of the dependence of the level number n of the
educational activity independence of a testee on the task number i. The graph analysis
shows that the level of independence increased monotonically to level 10, which
corresponded to the autonomous activity of a testee. However, a testee was not ready
for autonomous educational activities and as a result of this bifurcation, there was a
regression of educational activities, and a corresponding decrease from the 10 to the
7th level of independence.
Fig. 2. Level n dependence on task number i
Figure 3 shows an actiogram of the activity of a testee on the task performance. The
actiogram represents the mismatch value change graph of the current and target states
of the problem solution [20]. The mismatch value between the current and target
problem solving states at the start time is 12.
Fig. 3. Actiogram or task solution search trajectory
The actiogram shows that when solving the problem, a testee first uses the trial and
error method, and spends a lot of time on each action, but this does not give a result.
Then, a testee proceeds to a systematic search, which consists in sorting out all
options until there is the right action. This leads to a loss of the time pace of the task
and an increase in labor intensity. A testee spent more than 300 seconds to find a
solution to the problem.
3 Results
Quantitative processing of the time series of educational actions allows making a
numerical assessment of the information received by a testee from the reinforcements
of his actions. By searching for a solution to the problem, the student receives after
each action reinforcements that carries information I . A numerical evaluation I
can be obtained using the Shannon-Claude formula
n1 n n n
I 1 Н 1 log 2 1 2 log 2 (1)
n n n n
Where n n1 n2 - the total number of actions taken to solve the current task ,
n1 number of right actions, n2 number of wrong actions. In formula (1), instead
of the probability of choosing the right and wrong actions, the relative frequency of
the corresponding actions is used. Figure 4 shows graphs of entropy H dependence on
time t.
a) b) c)
Fig. 4. Dependence of entropy H of educational activity on time t for three
levels of the independence of students: low – а); middle – b); high - c)
4 Conclusions
The maximum entropy value is 1 and the minimum entropy value is 0. The graph
in Figure 4 (a) shows that the entropy of the weak student's activities fluctuates
steadily near 1. Of the sample, the proportion of such testees is about 27%. They
usually use random or systematic search for the right solutions. Without flexibility of
thinking and sufficient operative memory, a testee cannot solve the problems of facet
classification of English language tenses without constant reliance on external
information reinforcements from the problem environment. Such testees may be
expected to lack foreign language education [21].
Testees who successfully solve the problems of facet classification of English
language tenses, but cannot switch to autonomous activity mode, are represented by a
characteristic graph of entropy dependence on time in Figure 4 (b). For such testees,
the conditions of educational activity are more energetically comfortable, when it is
possible to make mistakes and compensate for them with external reinforcements.
The proportion of such testees in this sample is 41%.
Testees who reach zero entropy of educational activities (see Figure 4 c) have
abilities for autonomous educational activities. They are characterized by thinking
flexibility, developed operational and long-term memory. The percentage of such
testees is 32%. They achieve the state of autonomous educational activity, moving
from finding a solution by trial and error to intelligent decision-making.
Thus, the qualimetric approach to the dynamic evaluation of learning to solve the
problems of facet classification of English language tenses allows you to obtain
meaningful diagnostic information about the procedural characteristics of the
educational activities of the testees.
References
1. Safargaliev, E. R., Eremina, I. I., Savitsky, S. K., Camelina, V. A.: Mathematical Model
and Qualimetric Assessment of Graduate Education Quality in Environment Saturated
with Information and Communication Technologies [in Russian]. International Education
Studies 8(2), 78-83(2015)
doi:10.5539/ies.v8n2p78 https://www.elibrary.ru/item.asp?id=12853864
2. Lantolf, J.P., Poehner, M.E.: Dynamic assessment in the classroom Vygotskian praxis for
second language development. Language Teaching Research. 2010. 15(1) 11–33
https://www.researchgate.net/publication/274494809
3. Vygotsky, L.S.: Mind in society: The development of higher psychological processes.
Cambridge, MA: Harvard University Press (1978)
4. Meyers, J.: The training of dynamic assessors. In C.S. Lidz (Ed.), Dynamic assessment:
An interactional approach to evaluating learning potential. New York: Guilford Press, 403-
425 (1987)
5. Luger, George F.: Artificial Intelligence: Strategies and Methods for Solving Complex
Problems, 4th ed., M.: Williams Publishing House (2003)
6. Sutton, R.S., Barto, E.G.: Reinforcement learning. M.: BINOM. Knowledge laboratory
(2014)
7. Haywood, C., Tzuriel, D.: Interactive assessment. New York: Springer (1992)
8. Jacobs, E. L.: The effects of adding dynamic assessment components to a computerized
preschool language screening test. Communication Disorders Quarterly 22(4), 217-226
(2001) URL: http://dx.doi.org/10.1177/152574010102200407
9. Hwang, G. J., Panjaburee, P., Triampo, W., Shih, B.Y.: A Group decision approach to
developing concept – effect models for diagnosing student learning problems in
mathematics. British Journal of Educational Technology 44(3), 453-468 (2013)
https://dialnet.unirioja.es/servlet/articulo?codigo=6498727
10. Poehner, M.E., Zhang, J., Xiaofei, Lu.: Computerized dynamic assessment (C-DA):
Diagnosing L2 development according to learner responsiveness to mediation. Language
Testing 32(3), 337-357 (2015)
11. Poehner, M. E., Lantolf, J.P.: Dynamic Assessment of L2 Development: Bringing the
Pastinto the Future. CALPER Working Papers Series, 1(1), 1-26 (2019) doi:
10.1558/japl.v1i1.647
12. Vishnyakova, S.M.: Vocational education: Dictionary. Key concepts, terms, topical
vocabulary. M.: NMC ACT, (1999)
13. Zentsova I.M. Application of the facet method to the construction of the classification of
courses by choice (on the example of the discipline of physics) [in Russian]. Bulletin of
KSU after N.A. Nekrasov. 16(4), 22-27 (2010)
14. Dyachuk, P.P., Shkerina, L.V., Shadrin, I.V., Peregudova, I.P.: Dynamic adaptive testing
as a way of self-study of students in the electronic problem environment of mathematical
objects [in Russian]. Bulletin of Krasnoyarsk State Pedagogical University after V.P.
Astafiev. 1(43), 48-59 (2018) https://www.elibrary.ru/item.asp?id=32627011
15. Hegenhan, B.: Theories of Learning. 6th ed, M. Olson. St. Petersburg: Peter (2004)
16. Quastler, H.: Emergence of biological organization. Moscow: Mir (1967)
17. Lotman, Yu.: Dynamic model of a semiotic system. Moscow (1974)
18. Dyachuk, P.P. (Jr.), Dyachuk, P.P., Karabalykov, S.A., Shadrin, I.V.: Diagnosis of
unstable cognitive states of active agents [in Russian]. In the collection:
NEIROINFORMATIKA-2016. 259-270 (2016)
https://www.elibrary.ru/item.asp?id=27687397
19. Dyachuk, P.P., Shadrin, I.V., Peregudova, I.P.: Dynamic computer test simulator "English
Language Tenses" Certificate of Registration of the Program for Computers RU
2020663077, 12.10.2020. Application No. 2020662147 dated 22. 10. 2020
20. Dyachuk, P.P., Shadrin, I.V., Kudryavtsev, V.S.: Actiogram method in control systems
and diagnostics of human activity [in Russian]. B. Sb.: Systematic analysis and
information technologies "SAIT-2013" works of the V. International Conference.
Department of Nanotechnology and Information Technologies of the Russian Academy of
Sciences and others 121-128 (2013) https://www.elibrary.ru/item.asp?id=23546242
21. Rotenberg, V.S., Arshansky, V.V.: Search activity and adaptation. Moscow: Nauka (1982)