A Way to Automate the Preparation of Distractors for Test Questions in a Text-based Game Web Simulator for Training of Algorithmic Skills of Students

Abstract

One of the least time-consuming ways to organize effective automated training of algorithmic skills is a text-based game simulator, where the interaction is organized as in early computer games of “adventure” type. For each game state, the studied algorithm defined uniquely the correct answer - the next action. А question-and-answer interface, where a student makes a choice of an answer to a test question at each step, requires a lot of incorrect answers (distractors). We propose to use the formulations of answers that are correct for other states (vertices in the algorithm graph) as distractors, as well as use the correct answers from other algorithms in the same subject area. A method of preparing of such distractors set where the part of them then randomly selected for the user is described.

Keywords

• medical algorithms
• distractors
• game-based educations

Introduction

In the training process of future doctors at the Medical Academy, students study a large number of methods of treatment and prevention of various diseases. In addition to studying theory, learning also includes the development of practical skills. At the undergraduate, much attention is paid to training “by the patient’s bed”. One of the tasks facing a medical education is the need to narrow the gap between student theory and clinical practice [ 1 ].

The number of students’ practical skills that should be mastered by a future doctor is quite large. The implementation of the first stage of the medical reform of healthcare in Ukraine has led, for example, to the fact that primary care physicians (family doctors) must now possess the skills of both a pediatrician and a therapist (a general practitioner). Unfortunately, it is impossible to master all the necessary practical skills in a qualitative manner during 6 years of study at a medical university. For this, a postgraduate stage of study, internship and residency are provided.

Educational simulation centers have been created to improve the practical training of future doctors in recent decades, it allows you to train practical skills using models - mannequins, phantoms, computer applications and simulators.

Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

The main goal of our work is to develop a methodology for creating a virtual training simulator. The first subject area where we intend to implement the simulator was selected skills for providing first aid treatment. This choice is determined not only by the special importance of first aid treatment, but also by the high need of a large number of people who possess these skills.

According to the statistics, more than 2 million people die in Ukraine annually, 75% of deaths occur outside the hospital [ 2 ]. In Europe, 55-113 cases of sudden death are recorded per 100,000 people per year, which are 350-700 thousand cases per year [ 3-5 ]. Most deaths occur at home, at work, on the street and in other places before the arrival of an emergency doctor or paramedics. A significant part of these people could have been alive if they had been given first aid in time. Now, despite the fact that “the basis of medical knowledge” subject is included in the bachelor curriculum of most specialties, and the number of the population with higher education is steadily increasing throughout the world, the effectiveness of first aid treatment with the use of general medical knowledge of an ordinary person (without medical education) is extremely low [ 6 ]. 2

Training simulators

A training simulator should provide an opportunity to recreation of a situation when a person needs medical assistance realistically. In our case, the simulator should allow: ─ repeating the algorithm for providing first aid treatment repeatedly; ─ developing first aid treatment practical skills; ─ contributing to the formation of professional skills of students without the presence of patients; ─ minimizing the time of first aid; ─ maintaining professional skills for doctors in accordance with the qualification requirements (for example, cardiopulmonary resuscitation).

An important advantage of simulation training is the ability to receive feedback. Unlike traditional sources of knowledge (textbooks, videos), which do not provide feedback, simulators imply certain involvement of participants [ 7 ].

There are many classifications of educational simulators - proposed by Miller M.D. (1987), Meller G. (1997), Issenberg S.B. (2001), D. Gaba (2004) and Guillaume Alinier (2007), Gorshkov M.D., Fedorov A.V. [ 8-13 ]. According to the classification of D. Gaba, the virtual training simulator which we need belongs to the fourth group of simulators: “patients-on-screen” - simulators with a patient’s video image on the screen without using physical simulation, namely a computer simulator [ 11 ]. In another classification proposed by Professor G. Aligne, a computer simulator belongs to the second group - “products with a screen” [ 12 ]. According to the level (layer) of realism, there are seven layers of realism: visual, tactile, reactive (Low-Fidelity), automated, hardware, interactive and integrated (High Fidelity) [ 13 ]. Our project in the first iteration certainly refers to Low-Fidelity. Adding video animations and moving to a “virtual reality” environment will allow us to speak in the future about the level of realism of the High Fidelity simulator 3

Simulator subject area

We have chosen 29 algorithms for providing first aid treatment to people in emergency conditions for the first implementation of the subject area of skills [ 14 ]. In this paper, we restrict ourselves to considering only one algorithm from this set - cardiopulmonary resuscitation (CPR). This algorithm approved by order of the Ministry of Health of Ukraine and available online [ 14 ].

Mortality caused by problems of the circulatory system organs functioning stably takes the first place among all causes of death. Moreover, at the prehospital stage, that is, during the first hour, more than 50% of all those people die from acute myocardial infarction die (up to 85% of cases, the cause of death is ventricular fibrillation). However, in every fourth case, the body was viable, healthy and had no problems with the circulatory system. Cardiopulmonary resuscitation is an actively developing medicine area of critical conditions [ 2 ]. There is only 5 minutes to save a person’s life when a sudden blood circulation stop happens. But the likelihood that during this time an ambulance doctor will be in place is very low.

Thus, in order to save life in case of sudden blood circulation stop, everyone should be able to provide first aid. Statistics show that life often depends on how quickly and efficiently first aid is provided to the victim; this is called the rule of the “golden hour” - the time when a person is balancing on the verge of life and death, and when the victim can be provided with the most effective first aid assistance [ 15, 16 ]. The quality of CPR is the most important survival factor, the foundation of effective resuscitation. That is, the quality of CPR is a matter of life and death [ 17 ]. The faster and more qualitatively CPR is performed, the less time it will take to recover the body [ 2, 17 ]. 4

Text game simulator

The specific task of simulation training is to train the student to execute the algorithms of first aid correctly. Initially, these algorithms are given to us in the text form. Therefore, it is necessary to transform them into a different, graphical representation [ 18 ]. We believe that the UML modeling language is clearly redundant for these purposes, and we will depict the algorithms in the form of so-called flowcharts, according to the international standard ISO 5807-85. Fig. 1 shows a fragment of a graphical representation of the cardiopulmonary resuscitation algorithm.

The transition from texts to graphical diagrams of algorithms often shows the presence of ambiguities in the original text description, which require clarification. It is clear that the correct and unambiguous presentation of the algorithm does not guarantee its correct execution. Now the cause of incorrect execution is the actions of the performer. The task of executing the algorithm correctly is a typical “complex open ended assignments”. A general classification of errors that performers of complex open ended assignments can make is given in [ 19 ].

To implement our system, a software shell will be prepared that will be accessible to the user through a web interface. Correct execution of the algorithm is achieved through a sequence of choices of correct answers - first aid actions, according to the approved algorithm. The correct choice is determined by the condition of the patient (location in the graph of the algorithm). In the first iteration of the project, all the initial information, which is necessary for making the correct choice, is displayed on the screen in the form of text, possibly with graphic illustrations. That is, technically, it is going to be a hypertext page tied to a specific state (the step of the algorithm). Each answer is a hyperlink. Making the correct choice, this goes to the next state page, step of the algorithm. If you make incorrect choice, this goes to the page with an error message (patient's death).

Thus, we intend to transform each skill algorithm from our subject area into a textbased “adventure game”. The constructing technique of such text games is well known. In the days of text-based computer terminals and at the dawn of mass computerization, computers did not have the technical capabilities to form high-quality dynamic graphics yet. At that time, such computer games like “adventure” were very popular. Now we are creating a modern analogue of such games for educational purposes. Repeating the evolution of computer games, we also plan to move from static hypertext pages to computer graphics with high detail in the future.

At each step of the algorithm, for all acceptable at this step variants of the values of the key variable of the human health state (which usually can be no more than two), we can easily formulate the question of further action and the correct answer to it by the original text entry of the algorithm. Next, we should ask a set of other incorrect answers to each of the questions by the steps of the algorithm. Such deliberately incorrect answers are usually called distractors. 5

Distractor Preparation

A student who does not know the correct answer usually tries to guess it. Test developers should minimize guessing based on indirect evidence. As such a correction for guessing, one of the parameters in the three-parameter theory IRT (Item Response Theory) appears - the multiple-choice model [ 20 ]. The main subject of IRT models using is the assessment of the correct answer probability of test takers doing the tasks of various difficulties. The IRT does not analyze the sum of the test scores of test takers, but it analyzes the scores obtained for each task [ 21 ]. Another way to reduce the probability of guessing is to increase the number of distractors [ 22 ].

The function of distractors is to make test takers think actively in order to distinguish the correct answer from the plausible one [ 23, 24 ]. Unfortunately, now there is no unified theory of the distractors development, all recommendations are quite general in nature [ 25 ]. Prepared versions of incorrect answers should be plausible and, if it is possible, based on mistakes that students do most often. This should help consolidate the learned material and bring the action to automatism. In [ 26 ], a formal criterion is used - the frequency of a real choice of each incorrect answer. If you choose the correct selection of the distractor, the probability of its choice is close to random, which for the five answers is 20%. Distractors are considered acceptable if at testing they are selected by at least 5% of the test takers [ 27 ].

To determine the best distractors, it is necessary to collect and analyze data of their using results for a long time. The distractor should be equally attractive to students who do not know the correct answer [ 28 ]. Distractors must correspond to the subject area of the question, be formulated clearly and be both the wrong and plausible answer [ 28, 29 ]. The distractor must be homogeneous — it must belong to the same category as the correct answer [ 30 ].

We intend to develop many incorrect answers based on our classification of errors [ 31 ]. We believe that the most plausible distractors will be actions whose execution is provided in other parts of the algorithm. This corresponds to error 2.2 “error of transitions between actions - an incorrect transition was done, which is not on the algorithm diagram” of our classification. If the corresponding block is located on the sequential section of the algorithm over one block from the current one, then we have error 2.1.1. “Skipping of a single algorithm action” of our classification. The compilation of such distractors does not require any additional effort.

Error 2.1.2. “The execution of an incorrect action which is not provided by the algorithm” can be ensured in the same way if we use actions from other algorithms as distractors. It is necessary to make sure that they do not coincide with the distractors of this algorithm., Error 2.1.4 “Choosing the wrong branch (the conditional statement exit)” should be considered before each branch block of our algorithm.

Error 2.1.3. “Incomplete execution of the right action” can be implemented for complex operations of the algorithm, which consist of several simple actions. Here the urgent problem is the ambiguity of the recording of such complex operations of the algorithm, therefore we plan not to formulate such distractors as much as possible. There are practically no complex logical conditions in this set of algorithms, therefore, we also do not intend to model error 2.1.5. “Logical error in the difficult conditions”. 6

Distractor Selection Examples

We number all possible states - places of the flowchart of the algorithm in Fig. 1. For each numbered state, we set a text description that will be given to the student. Then we determine the correct answers - the following actions specified in the algorithm.

So for state 3, the text description will be “the victim is responding”, and the correct action is “checking if there are threats to the victim”. For state 8: description “the victim does not respond,” the action is “seek help from others”. For state 10: description - “the victim lies on his stomach”, the action is “turn the victim on his back”. For state 11: description - “now the victim is lying on his back”, the action is “to restore airway patency”. We may need to give a more complete description of the condition indicating the data collected earlier (in particular, the results of the selection — checking the conditions of the logical operator), then, for example, state 10 will be described as follows: “the victim does not respond, the victim lies on his stomach”. It is possible that a more convenient way to describe the current state will be a list of all previous actions and checks of logical conditions. Then it will be possible to simulate a situation when the test taker has joined the first aid process, when the part of the first aid actions have already been completed.

In determining the set of suitable distractors based on the correct answers for other states, we should exclude those states that we have already visited from consideration. That is, for state 11, we should not include actions from states 1,2,8,9,10 in many suitable distractors. The way of reaching many states is possible in several ways thus, state 12 can be reached through 10 and 11, or via 16, therefore, in the general case, we must exclude distractors obtained through all possible paths. So in the states located closer to the completion of the algorithm, the choice of distractors will be less.

Let the number of answer options for each test question in our system be five: the correct answer and four incorrect answers - the distractors. We will take half (two) of them from the correct answers for other states of this algorithm, and the other half from the correct answers for other algorithms. We will exclude duplicate options that correspond to the same states that differ only in their achievement way.

The selecting of distractors set for each specific training session will be implemented by using a random number generator. So for state 10: to the correct answer “turn the victim on his back”, for example, such distractors can be added: 4 “move the victim to the safe place”; 12 “determine the presence of breathing”; 13-c “place clothes / pillow under the victim’s head”; 19-10 “cover the victim with a blanket (or thermal blanket)”; This particular set of distractors was obtained by sorting of two lists randomly. As a number of actions are the same in many first aid treatment algorithms, these repetitions must be eliminated before making a random choice. The answer options are also arranged in random order when they display on the screen.

There steps of the algorithm (questions) in text game will be placed on the various hypertext page. The transition between the blocks of the algorithm (from one question to another) is carried out due to the transition to another HTML pages via hyperlinks.

The future steps of this project are dynamically generated HTML pages with a question and a set of options and translation of the content to other languages (English, French, and Russian). 7

Conclusion

Distractors compiling is one of the most laborious tasks in preparing of test questions. A set of questions and correct answers are set by the algorithm of the action that the student is learning. We use answers that are correct in other states (points of the algorithm) as distractors for questions formulated for other states. In this case, we exclude those answers that have already been selected in the current session – during the way according to the algorithm. The randomization of the remaining distractors provides a set of plausible alternatives. After receiving data on the frequency of distractor choosing by students, it will be possible to make changes in the frequency of occurrence of one or another distractor.

References

1. M. M. Korda , A. H. Shulhai , A. A. Hudyma , S. Y. Zaporozhan , Devolopment of practice - oriented and simulation traning at I. Horbachevsky ternopil stste medical university (in Ukrainian) . Medical Education , 2 , 54 - 57 ( 2016 ). doi: 10 .11603/me.v0i2. 6214 .

2. Yu . Dumansky, N. Kabanova , I. Verkhuletsky, N. Sinepupov , A. Osipov , Serdechnolegochnaya i cerebralnaya reanimaciya (in Russian) ., Med . Neotl. Sost., 1 ( 40 ), 138 - 147 ( 2012 ).

3. J. Berdowski , R. Berg , J. Tijssen , R. Koster , Global incidences of out-of-hospital cardiac arrest and survival rates: systematic review of 67 prospective studies . Resus ., 81 ( 11 ), 1479 - 1487 ( 2010 ). doi: 10 .1016/j.resuscitation. 2010 . 08 .006.

4. J. Gräsner , J. Herlitz , R. Koster , F. Rosell-Ortiz , L. Stamatakis , Quality management in resuscitation-towards a European cardiac arrest registry (EuReCa) . Resus. , 82 ( 8 ), 989 - 994 ( 2011 ). doi: 10 .1016/j.resuscitation. 2011 . 02 .047

5. J. Gräsner , Epidemiology and management of cardiac arrest: what registries are revealing . B. pract. & res. Clin. Anaesth ., 27 ( 3 ), 293 - 306 ( 2013 ). doi: 10 .1016/j.bpa. 2013 . 07 .008.

6. S.A. Boytsov , P.V. Ipatov , A.V. Krotov , Pre-hospital management role in cardiovascular mortality and lethality reduction (in Russian) . Cardiovascular Therapy and Prevention , 6 ( 4 ), 86 - 95 ( 2007 ).

7. K.B. Kurmangaliyev , A.S. Mukanova , G. Ye. Kuldanova , Sh.Zh. Salikova, M.I. Turebayeva , The review of the simulation methods of training in clinical preparation (in Russian) . Med. j. of West Kazakhstan , 1 - 2 ( 38 ), 80 - 84 ( 2013 ).

8. G. Miller , The assessment of clinical skills/competence/performance . Acad. Med ., 65 ( 9 ), 63 - 67 ( 1990 ).

9. G. Meller , A typology of simulators for medical education . J. Dig. Imag. , 10 , 194 - 196 ( 1997 ). doi: 10 .1007/BF03168699

10. B. Issenberg , М. Gordon, D. Gordon , R. Safford , Simulation and new learning technologies , Med. T., 23 ( 1 ), 16 - 23 ( 2001 ). doi:10.1080/01421590020007324

11. D. Gaba , Training and nontechnical skills: the politics of terminology, Sim . Health., 6 ( 1 ), 8 - 10 ( 2011 ). doi: 10 .1097/SIH.0b013e31820f9a55

12. G. Alinier, A typology of educationally focused medical simulation tools . Med.T., 29 ( 8 ), 243 - 250 ( 2007 ). doi: 10 .1080/01421590701551185.

13. M. Gorshkov , A. Fedorov , Classification of the Simulation Equipment (in Russian) , Virtual Simulators in Medicine , 1 ( 7 ), 21 - 30 ( 2012 )

14. Order of the Ministry of Health of Ukraine No. 398 dated 16.06 . 2014 . “ On approval of procedures for providing pre - medical assistance to persons in emergency situations” (in Ukrainian) https://zakon .rada.gov.ua/laws/show/z0750- 14 .

15. G. G. Roshchin , V. O. Krylyuk , V. P. Pechyborshch , I. O. Yosypenko , Trends of emergency department organization in general hospitals in the healchcare system of the UK and the USA (in Ukrainian) , Surgery of Ukraine. 4 , 114 - 122 ( 2014 ).

16. A. Vlasov , N. Ryzhova , The main exam on propaedeutics - the “golden hour” in the work of the ambulance paramedic (in Russian). A Paper presented at the International Scientific and Practical Conference, Magnitogorsk, 11 October 2018

17. V. Vechyora , A. Strelkov , Obuchenie naseleniya pervoj medicinskoj pomoshi s ispolzovaniem simulyacionnogo oborudovaniya . Paper presented at the Conference “ Simulation technologies in the training of medical workers: relevance, problematic implementation and prospects (in Russian) , Grodno, 7 December 2018

18. M. Kopatskaya , O. Khorolsky , Y. Zhykharieva , A. Dubinsky. The standardization of the graphical representation of medical algorithms (in Russian) . Presented at the XI International Scientific Conference “Mathematical and software of Intelligent Systems (MPZIS2013)” , pp. 117 - 118 ( 2013 ).

19. V. Latypova , A. Tyurganov , Metodika upravleniya obucheniem na osnove klassifikacii oshibok v slozhnyh otkrytyh zadachah (in Russian) . Engineering journal of Don , 4 ( 47 ), ( 2017 ).

20. T. Lisova, Tendency to guess the right answer to a test question: The gender aspect, Presented at the Conference “Empirical Studies for Educational Reform in Ukraine” (in Ukrainian) ( 2017 ).

21. А .V. Rodionov , V.V. Bratischenko , Application IRT - model for the analysis training results within the competence approach (in Russian) . Modern problems of science and education, (4) , 150 - 150 ( 2014 ).

22. E. Girfanova, Minimizing guessing of distractors when compiling test questions on Russian as a foreign language (in Russian) . Presented at the XXIII conf. “The study and teaching of Russian literature in the era of linguistic globalization ” p. 157 ( 2018 ).

23. O. Kvasova , Fundamentals of Testing of Foreign Language Skills and Skills: A Textbook (in Ukrainian) , Kiev: Lenwith ( 2009 )

24. K. Lastovka , The organization of an English entrance test to MA course: lingual and didactic aspect . Educological discourse (in Ukrainian) , 2 ( 6 ), 97 - 109 ( 2014 ).

25. I. Shmarovoz, Characteristics of multiple choice technique. Advantages and disadvantages (in Ukrainian) . Collection of scientific articles of teachers and students of the faculty foreign languages Nizhin: Publishing house of NDU M. Gogol , 121 ( 2012 ) http://lib.ndu.edu.ua:8080/dspace/bitstream/123456789/201/1/125_Zbirnik_ naukovih_praz c . PDF

26. I. Belonogov, Methodology for the development and compilation of control programs in military medical disciplines and the generalization of experience in their use “Military medicine” (in Russian), 1 , ( 2011 ).

27. V. Avanesov , Pedagogical measurements by the George Rush model (in Russian) , J of School Technology , 4 , 158 - 172 ( 2010 ).

28. A .V Mogilev (ed.), Technologies for searching and storing information (in Russian) . Control Automation Technologies (BHV-Peterburg , 2012 ).

29. O. Lagodinsky Methodical bases of testing for the control of foreign language listening and reading system (in Ukrainian) , Presented at the Conference “ Modern concepts and tendencies of teaching a foreign language in the global world ” KNEU, Kiev, 5 Oct 2010 . https://ir.kneu.edu.ua/handle/2010/15338

30. S. M. Case , D. B. Svensson . Creation of written test questions on basic and clinical disciplines . (Philadelphia: National council of Medical Examiners.) 119 pages ( 1996 ).

31. Y. Zhykharieva , A. Dubinsky , Classification of user errors when performing the medical algorithm, (in Ukrainian) Presented at the Conference “Information technologies in the educational process 2019” https://drive .google.com/file/d/1DASEiW6KKHLtOIhyzpq3EsoAlhQsnPEy/view