Problem of design and implementation of testing tools of learning management system which make use of virtual and augmented reality objects for training of cybersecurity professionals is discussed. An approach for constructing the Augmented Reality enhanced tests within Learning Management System is proposed. Testing data from cybersecurity curriculum are processed to identify use-cases and learning problems for which implementing Extended Reality elements will bring the greatest benefit for competencies forming. Restrictions for wide use of Augmented Reality in cybersecurity trainings are discussed. Learning scenario, augmented reality, leaning management system, cybersecurity The pandemic and related quarantine restrictions have significantly accelerated the digitalization of many industries. Role of digital twins [1-3] in e-commerce, education and industry is growing nowadays. The rapid development of computer-aided design, virtual reality (VR) and augmented reality (AR) technologies [4] allows the creation of realistic copies and 3D models of various objects. The current level of technology and the state of the market provide the conditions for the implementation of VR and AR applications to support different methods of input and output (text, audio, video, AR / VR, holography, etc.). As hardware and software both evolve over time, new technologies to facilitate human-machine interaction must be adopted for augmented reality technologies to enhance the collaboration experience. The variety and ubiquity of IoT devices, cameras, sensors and RFIDs allow real-time and very accurate tracking, which makes the integration of software applications with smart home positioning systems and other cyberphysical systems very attractive.
2022 Copyright for this paper by its authors. augmented reality, as well as web-3D will achieve a new level of user interaction with the content of education.
Virtual Reality (VR) is a completely immersive virtual environment which substitutes the real environment in every aspect – these are on the opposite sides of the so-called Milgram’s continuum. Augmented reality (AR) is a kind of superposition of these extrema, adding a digital layer with virtual object or information into the picture of a real environment. Extended reality (XR) has the important feature that the real world and the user interact with the digital (virtual) layer in real time. Using the large variety of IoT sensors available today, this human-machine interaction can be activated and mediated by touch triggers in an intuitive and inclusive way. It is generally agreed that advanced technologies of shared telepresence can help solve the problems of e-learning associated with the lack of personal contacts and intuitive modes of interaction.
Among a variety of instructional design strategies for e-learning courses, immersive virtual elements gain popularity. These can substantially enhance Scenario Based Learning (SBL) which is widely used to meet varied corporate training needs such as soft skills training, professional skills training, application simulations training and compliance training in industry [9, 10]. In a virtual environment, restrictions of spatial separation and disabilities become irrelevant to the learning scenario realization. Mixed reality components can be embedded into LMS to substitute on-site training [11-15], when training facilities are not available. In addition, by using context-aware XR system for instructional design, one may eliminate the majority of human errors, unavoidable in the standard step-by-step operational guides, simultaneously collecting learning statistics and psychophysiological indicators. The possibility to provide important competences and relevant experience in a natural, everyday-like environment must be taken into account first. Then, the realizability of the standard SBL design procedure is to be considered and, finally, availability of a technological platform for developing and displaying XR elements in leaning management system (LMS) is to be guaranteed. For instructional design, SBL may become a method of choice when a decision made by a learner in a certain moment will determine further course of actions (causality principle), analogous actions in similar situation will produce similar effects (resemblance principle) and textual information can be optimally combined with visual one (contiguity principles). SBL is the only viable solution if provision of the real-world experience is too costly or dangerous, requires time compression to show the long-term effects of decisions and actions or multiple trial-and-error experiences are to be provided to the learner to achieve the required competence. VR tools will be preferred when the real environment contains distractors, lacks the distractors which may be present in the modelled situation or does not allow to fully immerse the trainee into a stressed state of the actual decision-making situation. Cost of substituting real equipment with its digital twin, however, is low only if the model and it’s behaviour do not have to be very realistic, so the digital twins have some restrictions for use, too.
Complex SBL requires considerably more effort to design and implement as in many cases there exist multiple correct or acceptable solutions to the problem [16]. An important benefit of XR implementation in training is that a trainee uses not just his analytical abilities and problem-solving skills, but also enact his senses, uses and develops soft skills and receives a response resulting from his decisions in real time and through multiple channels of communication. Increase of the learner’s engagement is especially important for highly formalized routines such as mandatory compliance training which particularly comes from safety and security standards and frameworks.
The combination of problem-based learning, widely used today, with XR-enhanced scenario-based learning offers many benefits for both learners and instructions, providing the former with authentic experiences and relevant knowledge and the latter with quantitative educational data and tunable instructional tools. This paper suggests an approach for constructing the AR-enhanced tests within LMS increase the efficiency cybersecurity training. 2
Augmented reality technologies have been used in design, creative industries, education, and manufacturing since the 1990s [4,17,18]. The technology is to overlay a layer of virtual objects on the image of real objects in the environment. In this case, the internal structure, static and dynamic characteristics, etc. can be displayed in graphical or textual format. AR partially uses technological solutions, developed for virtual reality applications. Hardware user interfaces include traditional monitors and tablets, translucent windows, glasses, helmets, masks, and more. The most common today are tablets (handheld AR displays) due to high portability, improved cameras, displays, graphics chips, network equipment. These features of the current stage of implementation of augmented reality tools on e-commerce platforms create the preconditions for automating the cycle of development of AR / VR applications [19] for e-learning.
Majority of existing AR technologies create 2D images that are close but not perfect matches to the real object. The visual apparatus, observing 2D images, can give a false impression of the object, as is the case with anamorphic optical illusions. As a viable prototype we have tested a holographic projector designed to display 3D images of AR-models we have developed previously [7].
AR training platform sends digital information through interface device (e.g., HoloLens) so operators can receive information on how to interact with equipment and understand the more delicate parts of the procedure (Figure 1). VR training, on the other hand, offers a completely controllable environment where warning messages can be triggered by incorrect conduct. The goggles can also use Leap Motion tracking to register the hands of the user so they can interact with virtual elements without controllers or gloves. For full interaction, the AR system (usually a mobile client) must determine and monitor the position of the actor, on this basis to calculate and convert the visible image, to respond to control influences from the person. The interaction history can be stored in the cloud storage and processed to improve the service from compliance perspective according to the privacy and cybersecurity requirements.
The initial information for creating the technical task is a description of the object and the training scenario (photo or video of the object for which AR / VR means of display are required, description of behavior / interaction). This information must be provided initially by the course instructor. Technical features of a particular platform, specifications, standards, etc., about which the instructor may not have sufficient information and knowledge is to be specified by the analysts in collaboration with the developers. The analyst himself makes two types of decisions: on whether the information is sufficient to begin development and on the product compliance with standards and specifications. The functions of the information system to be developed are shown in Figure 2.
Important component of suggested information system is LMS which is an integrated system with a rather complex architecture to support the needs of teachers and students. Educators can use LMS to design materials and tests, communicate with students, and monitor their learning progress. Students can use it for learning and collaboration. LMSs provide logging, content access, planning, reporting, assessment, account management. For the purposes of this study, it is important that LMSs can also be used to track user actions. The last two levels of the LMS are designed to create, configure, and share service data with users. In general, an LMS can be divided into administration tools, teacher tools, and student tools. The system allows the student to discover his interests, inclinations and real skill what actually represent features for student model. While the student is using the system, various tools provide him with navigation, assessment, contextual help, and skill testing. Using available modern tools and frameworks for developing mixed reality assets, majority of real environment elements can be substituted by their virtual twins [11]. Input information for such instructional AR (VR) design is already present (curricula, use-cases, visuals) in, or may be collected (testing statistics, interaction logs) by learning management system. 3
The development of augmented reality tools requires considerable time and relevant resources, and also imposes certain requirements on end-user devices. Therefore, while improving online learning courses in LMS, a reasonable approach is to introduce elements of augmented reality gradually into those categories of tasks that, firstly, allow such improvement, and secondly, will give the greatest effect, expressed in improving the quality of students training.
TNTU uses LMS "Atutor", which allows to accumulate statistics on the success of each student, as well as statistics on test questions complexity. This data allows to design improvements to the test subsystem of LMS using augmented reality tools. We have already discussed the importance of AR and VR components in study process earlier. AR approach is particularly valuable in information security field which concern technical methods of information protection. The cost of devices and equipment used in the technical protection of information is extremely high and often not affordable for Ukrainian universities, so the use of AR technologies and XR components in the study of relevant subjects allows to partially compensate for the lack of the expensive equipment. For certain courses, such as "Systems of technical protection of information", visualizations allow to ensure the effect of presence and significantly bring the learning environment closer to professional tasks and real situations.
At the stage of test system development, it is important to construct test questions not only to test the knowledge level, but also to make student’s progress. LMS "Atutor" track statistics of students’ rating based on their performance in all disciplines, task complexity statistics as percentage of correct answers to all given answers and students’ success statistics within each test. But all these data are not really connected, because, for instance, question complexity can appear to be high just because low rating students mostly answer it. So, it is important to make a connection between student rating and question complexity. Authors of [9] suggest to use Rasch model for adaptive testing. We used this approach to assign level of knowledge and task complexity in order to identify the most difficult questions for AR supporting.
Rasch model can be represented by formula: =
1.7( − ) 1+ 1.7( − ) , (1) where is a parameter that represents student’s knowledge level and is a parameter that represents question complexity. is a probability to answer question of complexity by student with rating level.
In order to identify questions and tasks in which augmented reality tools should be included on the basis of statistical data on the success of passing tests by students, the following sequence of actions should be implemented:
1. Identify categories of questions that allow for quality improvement through the use of augmented reality tools.
2. Aggregate the data of passing by individual students.
3. Choose questions for which the number of passes is sufficient to ensure the reliability of the decisions made (in our case - more than 10).
4. Choose questions that, according to testing data, have an optimal resolution (close to 0.5). 5. Aggregate data by categories of questions (content modules). 6. To determine the most promising category for AR implementation.
Within online course "Systems of technical protection of information" we have determined test categories for which the creation of applications is possible, namely "Signs of embedded devices" (category 1), "Channels of information leakage" (category 2), "Physical foundations of information leakage" (category 3) and "Technical means for disclosing embedded devices" (category 4). For certain categories of questions, such as those related to regulatory documentation, the use of augmented reality tools is not appropriate at all. In some cases, when such an application can be implemented, it is associated with the creation and implementation of scenarios, which significantly increases the development time and requirements for the conditions of rational application, so it is not appropriate at the initial stage of course improvement.
We have split all students into three categories: underperforming (those, who have done less than 60% of test questions), best performing (top 10%) and normally performing (rest). We have also identified 10 categories of question complexity by splitting the total interval by 10 equal subintervals.
Each test category includes questions of different complexity, so it is not evident which category is better to start to enhance with AR.
Implementation of this algorithm yielded the results shown in the figures below. Here, probabilities of correct answer are visualized separately for students of different performance levels. For the current research, threshold values have been chosen manually based on the expert estimations, however, when there is sufficient statistical data and enough AR tools developed, one can automate this process and use our methodology to choose tasks for a particular student in adaptive way.
Figure 3 illustrates the distribution of probabilities to answer question correctly for each of three different categories of students and for all four chosen test categories. As we see the test category 1 stands apart the rest for all groups of students. It means that this topic is probably the most difficult to understand for all students and need to be improved with AR learning scenario.
To achieve substantial progress already in the first phase of AR-enhanced tests implementation and use optimally the propaedeutic function of the testing, it is advisable to begin developing and implementing extended reality elements for test questions and problems from category 1 ("Signs of embedded devices") and then for category 4 ("Technical means for disclosing embedded devices").
As pilot test questions it is reasonable to choose a few ones with optimal values of test complexity (close to 0.5). Introduction of pilot test questions with AR elements will allow both familiarizing students with AR elements in tests and improving their proficiency in constructing visual patterns for future professional activities. Only after thorough testing and analysis of the collected statistical data compared against the presently available base dataset, further steps in development and implementation AR assets are to be taken. Such a gradual approach for implementing AR in testing has many reasons. First, it is the high cost of AR asset design and development. Second reason is the inexpediency of overloading tests with visual tools which may cause latency increase for server-user communication. Thirdly, refinement of the AR models and test scenarios are to be done timely on the basis of analysis by academic staff and feedback from students. The last but not least factor is peculiar character of professional profile. Cybersecurity analyst primarily works with highly abstract numerical and graphical information, therefore, his analytical scills must be trained at the same types of instructional tests. After the development and testing of AR prototype test questions, the accumulated statistics and its analysis will allow to optimally design adaptive testing tools in which problems with or without AR will be combined according to the student profile, achievements, goals and desired competencies. Probability distribution of correct answers versus student performance for the analyzed dataset is shown in figure 4.
As one can see from figure 4, for target values of student performances the available data allows to build almost linear scaling of the testing data into the student mark, therefore it is a good starting point for implementing novel extended reality tools.
In this paper a possibility and a rationale for an implementation of extended reality tools into testing subsystem of a learning management system is studied. Using the data collected during one semester in course "Systems of technical protection of information", for a few categories of test questions a plausibility of choosing pilot test question and use cases is proven. Analysis of statistical data even for a limited cohort and relatively short testing period allows to determine which test questions and use cases would require an enhancement by implementing AR object to optimally improve the student performance.
[1] Chan Qiu, Shien Zhou, Zhenyu Liu, Qi Gao, Jianrong Tan, Digital assembly technology based on augmented reality and digital twins: a review. Virtual Reality & Intelligent Hardware 1 (2019) 597– 610.
[2] N. Maiellaro, A. Varasano, S. Capotorto, Digital Data, Virtual Tours, and 3D Models Integration Using an Open-Source Platform. In: Duguleană M., Carrozzino M., Gams M., Tanea I. (eds) VR Technologies in Cultural Heritage. VRTCH 2018. Communications in Computer and Information Science, vol 904. Springer, Cham, pp. 148–164. https://doi.org/10.1007/978-3-030-05819-7_12.
[3] Damiani L., Demartini M., Guizzi G., Revetria R., Tonelli F. Augmented and virtual reality applications in industrial systems: A qualitative review towards the industry 4.0 era. IFACPapersOnLine. 2018. Vol. 51. P. 624–630.
[4] R. Azuma, M. Billinghurst, G. Klinker. Special section on mobile augmented reality. Computers & Graphics 35 (2011) vii–viii.
[5] M. White, P. Petridis, F. Liarokapis, D. Pletinckx. Multimodal Mixed Reality Interfaces for Visualizing Digital Heritage, International Journal of Architectural Computing. 5 (2007) 322–337.
[6] M. Koehl, N. Brigand, Combination of virtual tours, 3d model and digital data in a 3d archaeological knowledge and information system, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., XXXIX-B4 (2012) 439–444.
[7] O. Kramar, Y. Drohobytskiy, Y. Skorenkyy, O. Rokitskyi, N. Kunanets, V. Pasichnyk, O. Matsiuk, Augmented Reality-assisted Cyber-Physical Systems of Smart University Campus , 2020 IEEE 15th International Conference on Computer Sciences and Information Technologies (CSIT) 2 (2020) 309-313.
[8] S. Gonizzi Barsanti, G. Caruso, L.L. Micoli, 3D Visualization of Cultural Heritage Artefacts with Virtual Reality devices, Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. XL-5/W7 (2015) 165–172.
[9] Tomashevskiy V., Pohrebniuk I., Kunanets N., Pasichnyk V., Veretennikova N. (2021) Construction of Individual Learning Scenarios. In: Hu Z., Petoukhov S., Dychka I., He M. (eds) Advances in Computer Science for Engineering and Education III. ICCSEEA 2020. Advances in Intelligent Systems and Computing, vol 1247. Springer, Cham. https://doi.org/10.1007/978-3-03055506-1_54 pp. 609–620
[10] V. Parson, S. Bignell, Using Problem-Based Learning within 3D Virtual Worlds, Cutting-edge Technologies in Higher Education 4 (2011) 241-261. 10.1108/S2044-9968(2011)0000004014. www.researchgate.net/publication/235264735_Using_ProblemBased_Learning_within_3D_Virtual_Worlds
[11] Yu. Skorenkyy, R. Kozak, N. Zagorodna, O. Kramar, I. Baran, Use of augmented reality-enabled prototyping of cyber-physical systems for improving cyber-security education, J. Phys.: Conf. Ser. 1840 (2021) 012026.
[12] Gutiérrez-Carreón G., Daradoumis T., Jorba J. Integrating Learning Services in the Cloud. Journal of Educational Technology & Society 2015. Vol.18 (1). P.145-157
[13] Underwood J., Cavendish S., Dowling S., Fogelman K., Lawson T. Are integrated learning systems effective learning support tools? Computers & Education. 1996. V. 26. P. 33-40.
[14] Jianhui Chen, Jing Zhao. An Educational Data Mining Model for Supervision of Network Learning Process. International Journal of Emerging Technologies in Learning (iJET). 2017. Vol.13(11). P.67.
[15] Hixon E., Ralston-Berg P., Buckenmeyer J., Barczyk C. The Impact of Previous Online Course Experience On Students’ Perceptions of Quality. Online Learning. 2016.Vol. 20. P. 25–40.
[16] J. Good, K. Howland, L. Thackray, Problem-based learning spanning real and virtual words: a case study in Second Life, ALT-J 16:3 (2008) 163-172, DOI: 10.1080/09687760802526681 www.tandfonline.com/doi/full/10.1080/09687760802526681
[17] Bacca J., Baldiris S., Fabregat R., Graf S., Kinshuk. Augmented Reality Trends in Education: A Systematic Review of Research and Applications. Educational Technology& Society. 2014. Vol. 17. P.133–149.
[18] Santos M.E.C., Chen A., Taketomi T., Yamamoto G., Miyazaki J., Kato H. Augmented Reality Learning Experiences: Survey of Prototype Design and Evaluation. IEEE Transactions on Learning Technologies. 2014. Vol.7. P.38-56.
[19] E.Evans, Domain-driven Design: Tackling Complexity in the Heart of Software, AddisonWesley Professional, 2004.