Guided Experiential Learning (GEL) [1] is a pedagogical framework in which proficiency is gained through focused, repetitive practice under real world or simulated real world conditions. It is increasingly implemented with the aid of games, simulations, virtual, augmented, and mixed reality. In these implementations, data from learning environments is used in algorithms and AI models that evaluate performance, estimate learner states, and support instructors or adaptive instructional systems in selecting scenarios that provide learners with targeted practice under a targeted set of conditions. Underlying these implementations are multi-stage data strategies that rely heavily on data and data exchange standards, as is illustrated by the Synthetic Training Environment Experiential Learning - Readiness (STEEL-R) project [2, 3]. Relevant standards include standards published by IEEE, 1EdTech, W3C, and other standard development organizations as well as standards associated with the US Advanced Distributed Learning (ADL) Initiative's Total Learning Architecture [4, 5]. This paper outlines the STEELR data strategy as an example of a GEL implementation, categorizes relevant standards, and discusses how they are used in STEEL-R, and suggests possible future enhancements that will be needed both for AI-enabled GEL systems.
We start our discussion of standards for AI-based GEL by giving an overview of the Synthetic
Training Environment Experiential Learning – Readiness (STEEL-R) [
Underlying the functional perspective in Fig. 1 is a data strategy. In this strategy, the Generalized
Intelligent Framework for Tutoring (GIFT) [
As of the writing of this paper, STEEL-R uses deterministic methods for converting performance evaluations to assertions, using assertions to estimate competencies, and recommending training experiences. All of these functions, however, are designed with AI in mind and are anticipated to involve AI and ML in the next iteration. 1.3.
Turning to the main topic of this paper, multiple standards represent and communicate data in
STEEL-R (see Fig. 2). These include standards used in the Advanced Distributed Learning (ADL)
initiative’s Total Learning Architecture (TLA) [
Since circa 1997, many standards development organizations (SDOs) have produced standards
intended to support the development, deployment, and operation of learning technologies. Leading such
SDOs include the Aviation Industry Computer-Based Training Committee (AICC, closed after 25+
years of operation [
Standards that support general learning technologies apply to GEL systems, but in many cases require modifications, extensions, or new components. This is because most existing learning technology standards evolved from formal education and regulatory training. The standards that prevailed were, as is often the case, the ones that were most market-relevant, and in this case the market involved Learning Management Systems (LMSs) that delivered education and training in cognitive domains and assessed learners at a single point in time. GEL, in contrast, tends to require longitudinal assessment, multi-modal delivery, and non-cognitive competencies and skills. Richer data must be tracked, computational models may include components such as training conditions and records of practice, and multi-dimensional competencies and skills frameworks are in-volved. Moreover, as Artificial Intelligence (AI) and Machine Learning (ML) become more central to the operation of GEL systems, the properties of learning experiences that matter will shift from those that help human operators catalog them to those needed by AI-driven recommendation and sequencing engines. As we go through the list of most relevant learning technology standards, we will point out where and how changes should or may be made to support GEL. 2.1.
The most prevalent standards that track and report on student activities in learning systems are AICC Computer Managed Instruction (CMI) standards [20], the Shareable Object Reference Model (SCORM) [21], xAPI, and IMS Caliper Analytics (Caliper) [22]. The first two – AICC and SCORM – were LMS-centric standards designed to report what a student completed and the results of quizzes and tests. xAPI and Caliper replace these with standards that can be used for other types of reporting, but when deployed as replacements for AICC or SCORM, they are usually configured to report the same data, i.e., completions of exercises and learning units and the results of formative and summative assessments on cognitive tasks.
GEL requires systems to report performance on complex tasks and behaviors, not just formative and summative assessments. As explained in [23–25], xAPI can do this with the aid of properly designed xAPI profiles [26] that enable more varied sets of verbs and contexts to appear in xAPI statements, and Caliper can be extended in similar ways. Developing an appropriate xAPI profile was a key enabler of the STEEL-R data strategy, and it is likely that much of the GEL standardization efforts around tracking and reporting will focus on the development of profiles which themselves may be viewed as standards for a particular type of GEL system or application domain. If this comes to fruition, then registries of profiles will be needed. Such registries have been set up commercially [27] and by the ADL [28] but to the best of the author’s knowledge have not become commonplace or standardized. 2.2.
When AICC and SCORM were developed, learning experiences were selected, sequenced, and delivered by an LMS. Selection and sequencing could be governed by instructions contained in AICC and SCORM packages (see Section 2.3). In the world of AI-enabled intelligent tutoring systems (ITS) and, more generally, adaptive instructional systems (AIS) [29], progressions through topics and learning experiences are governed by algorithms that implement the outer loop [30], for which we know of no widely adopted standard. Nonetheless, the outer loop generally applies to a single ITS or AIS, whereas in STEEL-R and the training environments it is meant to support, relatively granular learning experiences are delivered by multiple systems, and outer loop style adaptation is accomplished by GIFT, which also can configure training as it is in progress. We propose that this process be called experience orchestration, or XO. xAPI and Caliper have moved beyond an LMS-centric view of learning technology by supporting decentralized networks of learning systems. They do not, however, ad-dress how experiences are selected or sequenced. In STEEL-R, XO instructions given to GIFT come from human operators. We believe this will soon be replaced enhanced or replaced by AI-driven XO. For this purpose, two capabilities will be necessary: 1. An interoperable means of expressing XO patterns or instructions so that systems such as GIFT can understand and execute them; and 2. A standardized way of expressing the properties of learning experiences, learning goals, and learners that the AI examines when making XO decisions.
With regard to the first capability, many researchers have proposed methods of representing learning paths [31–35], and a schema for expressing learning pathways was developed by the Credential Engine as part of its Credential Transparency Description Language (CTDL) [36]. These proposals, as well as the CTDL schema, operate at the level of courses and credentials and not at the more granular level of learning experiences. At the other end of the granularity spectrum, ITS sequence knowledge components (KCs) based on programmable instructions [37, 38], although as mentioned earlier not in any standardized manner. There seems to be a need for standards for XO instructions that operate at the level of learning experiences (or “micro-learning”) that are finer grained than courses and larger than the instruction associated with single KCs. In passing, we remark that this requirement is reminiscent of the attempts made circa 2010 to use Business Process Execution Language (BPEL) to instantiate IMS Learning Design [39–41] and that, to the best of our knowledge, did not enjoy real-world commercial adoption. In our view, a standards are still needed to represent experience orchestration rules that depend on the properties of learning experiences, learning goals, and learner models – the latter of which are discussed in Section 2.4.
With regard to the properties of learning experiences, standards for learning object metadata were developed by the IEEE, IMS Global, ISO/IEC JTC1 SC36, the Dublin Core Metadata Initiative [42], and others precisely for the purpose of identifying and communicating these properties in interoperable ways. The problem faced in applying these to GEL is identifying which properties should be expressed more than it is developing new ways to express them. Moreover, there is ongoing activity in this area. For example, the IEEE standard for learning object metadata is in the process of being up-dated [43] and was used in STEEL-R with extensions that allow properties such as training conditions, available stressors, and difficulty factors to be expressed. CTDL, mentioned earlier, is naturally concerned with the properties of credentials but includes an extensive set of properties of learning experiences from which credentials are obtained. Schema such as the Creative Work schema hosted by Schema.org [44] are easily extended to include properties relevant to learning applications, as was done by the Learning Resource Metadata Initiative (LRMI) [45]. 2.3.
Packaging standards such as 1EdTech Common Cartridge [46] and SCORM manifests define content aggregations that can be loaded into an LMS or a learning environment. They identify the content to be delivered and include additional information, such as metadata describing the content and sequencing instructions. They are the analog of the recorded media that existed at the time those standards were developed and that could be inserted into a player and played, with a more modern analog being software containers [47].
For GEL applications, packages should include experience orchestration (see Section 2.2), and information about what will be practiced, how often it will be practiced, under what conditions it will be practiced, and how performance will be evaluated. For STEEL-R, researchers at the University of Texas at Austin developed an XTSP data format that represents experiences in synthetic learning environments. Earlier attempts at such representations, including IMS Learning Design [48], did not succeed in creating packages that could be delivered by multiple learning systems, but XTSP has promise. We expect that standardized abstract representations of learning experiences along the lines of XTSP will play a crucial role in GEL. 2.4.
A standardized interoperable form of a learner model [30] has long been a goal of the ITS/AIS community. This would allow any conformant AIS to read and update the data it uses to adapt learning to the needs of an individual and to export these data for use by the next AIS. The question this raises is what data are these?
A partial answer to this question is that the data needed to adapt learning to an individual’s needs is the learner’s competency profile, i.e., the list of competencies, skills, capabilities, traits, etc. possessed by a learner together with the level at which each one is possessed. For such a profile to be machineactionable, it must point to machine-actionable representations of competencies and competency frameworks, which is what CaSS provides as linked data in STEEL-R and in other implementations.
For GEL, snapshots of the competencies held by a learner are not sufficient. Since GEL requires that systems identify and deliver deliberate episodic practice at optimal intervals, learner models must have a time dimension, and quantities such as past practice should be included. The STEEL-R version of CaSS outputs competency profiles of this type and can associate values in user-defined concept schema to competencies. This presents a standardization opportunity that we believe would be of significant benefit to all AIS. 2.5.
A core notion in STEEL-R is that of an assertion. In STEEL-R these are stored and processed in CaSS, and to the best of our knowledge, the abstraction of an assertion in this form first appeared in CaSS. The term is now also used by 1EdTech in its Comprehensive Learner Record (CLR) specification [49], although in CLR assertions are about achievements, and achievements include accomplishments such as the completion of a degree or course as well as evidence of a competency. CaSS assertions are strictly about the possession or demonstration of a competency in an identified competency framework. Assertions are expressed in a standardized format within CaSS, but this format has not been standardized by any SDO.
An area where SDOs are actively involved is digital credentials and electronic learner records. Relevant standards include W3C verifiable credentials [50], Open Badges [51], Comprehensive Learner Records [49], and Learner and Employment Records [52]. These provide historical records that can be converted into competency assertions and combined with other evidence when estimating competency states. For use in GEL, credentials and records should include accurate timestamps and identify the type, conditions, and frequency of relevant practice, as is done for pilot licenses. This is necessary to create assertions of the type used in STEEL-R is not currently the case. It is not clear whether GEL will be considered as credentialing and badging standards mature, but we hope that it will be.
Finally, there are many non-learning technology standards that are likely to affect the design of GEL systems, especially if they incorporate AI. For example, the IEEE Standards Association has released several standards as part of its global initiative on the ethics of autonomous and intelligent systems [53], and is developing a related certification program [54]. It has also published a standard for an ageappropriate digital services framework [55], and around the world governments are passing legislation that affects data privacy rights [56] and the use of AI [57]. Standards and regulations of this nature are likely to proliferate in response to concerns about generative AI. In the GEL context, they are relevant to the design, development, and deployment of GEL systems and, regardless of their appropriateness, could present challenges to the ability of such systems to collect and process data.
Security is another area where regulations could affect the future of GEL. To quote from the EU Cyber Resilience Act (CRA) website [59], “From baby-monitors to smart-watches, products and software that contain a digital component are omnipresent in our daily lives. Less apparent to many users is the security risk such products and soft-ware may present.” The instrumentation used to collect data for GEL, including sensors and software (e.g., for virtual or mixed reality) will fall under the CRA, and in military and corporate settings, the systems used for GEL may be required to conform to standards such as the NIST 800-series standards [60].
Ignoring regulatory environments and concerns about issues such as privacy, AI ethics, and security is not a recipe for success, as was poignantly illustrated in the educational technology community by the collapse of In Bloom in 2014 [58]. We believe it is critical that the GEL community (and the educational and training technology in general) work with the government, non-governmental, and standards development organizations that are creating the regulatory and standards environments. GEL requires examining a leaner’s history, and GEL systems must collect more extensive data than most existing learning systems. This may make GEL systems more sensitive to privacy, ethical, and security concerns than more traditional learning systems. It is therefore important that the GEL community have a voice in the development of privacy, ethics and security standards and that they incorporate them into their own standards.
[19] CEN-WSLT (2014) LTSO - Learning Technology Standards Observatory :: In: CEN Workshop on Learning Technology. 0https://web.archive.org/web/20141114201940/http://www.cenltso.net/main.aspx?AspxAutoDetectCookieSupport=1. Accessed 2 Jul 2023 [20] AICC (1998) A980519C.pdf [21] ADL Sharable Content Object Reference Model (SCORM®). In: ADL Initiative.
https://adlnet.gov/past-projects/scorm/. Accessed 2 Jul 2023 [22] 1EdTech Caliper Analytics. https://www.imsglobal.org/activity/caliper. Accessed 2 Jul 2023 [23] Blake-Plock S, Hoyt W, Casey C (2021) Instrumenting GIFT with xAPI: a use case for IEEE P9274.3.x standards activity and implications for the broader field of ITS and AIS. In: Goldberg B (ed) Proceedings of the 9th Annual GIFT Users Symposium (GIFTSym9). U.S. Army Combat Capabilities Development Command – Soldier Center., pp 55–59 [24] Robson R, Ray F, Hernandez M, et al (2022) Mining Artificially Generated Data to Estimate
Competency [25] Blake-Plock S, Hoyt W, Casey C, Zapata-Rivera D (2020) Data Analytics and Visualization for xAPI Learning Data: Considerations for a GIFT Strategy. Design Recommendations for Intelligent Tutoring Systems 163–171 [26] ADL (2018) xAPI-profiles: A set of documents addressing the structure of and supporting services for xAPI Profiles. https://github.com/adlnet/xapi-profiles. Accessed 27 Feb 2022 [27] Rustici Software (2023) XAPI Registry. In: xapi.com. https://xapi.com/registry/. Accessed 3 Jul 2023 [28] ADL (2023) ADL xAPI Profile Server. https://profiles.adlnet.gov/. Accessed 3 Jul 2023 [29] Sottilare RA, Schwarz J (2020) Adaptive Instructional Systems. Springer International Publishing [30] VanLehn (2006) The behavior of tutoring systems. Int J Artif Intell Educ [31] Janssen J, Berlanga A, Vogten H, Koper R (2008) Towards a learning path specification.
International Journal of Continuing Engineering Education and Life Long Learning 18:77–97 [32] Ramos DB, Ramos IMM, Gasparini I, de Oliveira EHT (2021) A New Learning Path Model for
E-Learning Systems. IJDET 19:34–54 [33] Sanchez Nigenda R, Maya Padrón C, Martínez-Salazar I, Torres-Guerrero F. (2018) Design and evaluation of planning and mathematical models for generating learning paths. Comput Intell 34:821–838 [34] Salinas-Ibáñez J, De-Benito B (2020) Construction of personalized learning pathways through mixed methods. Comunicar 28:31–42 [35] Concentric Sky (2018) Open pathways for learning. In: 1EdTech.
http://www.imsglobal.org/concentric-sky-open-pathways. Accessed 3 Jul 2023 [36] Credential Engine (2022) New CTDL pathway constraints terms June release. In: Credential Engine. https://credentialengine.org/2022/07/06/new-ctdl-pathway-constraints-terms-junerelease/. Accessed 3 Jul 2023 [37] Corbett AT, Anderson JR (1994) Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model User-adapt Interact 4:253–278 [38] Aleven V, McLaren BM, Sewall J, Koedinger KR (2009) A new paradigm for intelligent tutoring systems: Example-tracing tutors. Int J Artif Intell Educ 19:105–154 [39] Chen C-T, Cheng YC, Hsu T-S, et al (2009) Delivering specification-based learning processes with service-oriented architecture: A process translation approach. https://citeseerx.ist.psu.edu/document?repid=rep1&type=pdf&doi=6f2606e6baf9 1d28497be73a7f53108f8b5ac93e. Accessed 4 Jul 2023 [40] Sandu V, Nedelcu A-V, Cocorada SA, Dinu S (2010) Instructional design in the vocational training on “computer networking.” In: 2010 12th International Conference on Optimization of Electrical and Electronic Equipment. ieeexplore.ieee.org, pp 1084–1090 [41] Kanagwa B, Agaba J, Tuheirwe D, Lunkuse S (2011) Improving Learning objects reusability through automatic generation web services [42] DCMI (2023) Dublin Core Metadata Initiative. https://www.dublincore.org/. Accessed 3 Jul 2023 [43] IEEE (2020) Standard for Learning Object Metadata [44] Schema.org CreativeWork - schema.org Type. https://schema.org/CreativeWork. Accessed 30 Mar 2020 [45] DCMI (2023) About LRMI. In: Dublin Core Metadata Initiative.
https://www.dublincore.org/about/lrmi/. Accessed 4 Jul 2023 [46] 1EdTech (2023) Common Cartridge. https://www.imsglobal.org/activity/common-cartridge.
Accessed 2 Jul 2023 [47] Wikipedia (2023) Containerization (computing). In: Wikipedia, The Free Encyclopedia.
https://en.wikipedia.org/w/index.php?title=Containerization_(computing)&oldid =1163448757 [48] Olivier B, Tattersall C (2005) The Learning Design Specification. In: Koper R, Tattersall C (eds) Learning Design: A Handbook on Modelling and Delivering Networked Education and Training.
Springer Berlin Heidelberg, Berlin, Heidelberg, pp 21–40 [49] Comprehensive Learner Record | IMS Global Learning Consortium.
https://www.imsglobal.org/activity/comprehensive-learner-record. Accessed 26 Apr 2020 [50] Sporny M, Longley D, Chadwick D (2022) Verifiable credentials data model v1.1. In: W3C
Verifiable Credentials. https://www.w3.org/TR/vc-data-model/. Accessed 13 Jun 2023 [51] 1EdTech (2023) Open Badges Home Page. In: Open Badges. https://openbadges.org/. Accessed 13 Jun 2023 [52] America’s R Learning and employment records, a pathway for. https://americafirstpolicy.com/assets/uploads/files/learning-and-employment-records-a-pathwayfor-restoring-americas-workforce.pdf. Accessed 12 Jan 2023