49 Exploring the Opportunities and Benefits of Standards for Adaptive Instructional Systems (AISs) Robert Sottilare1 [0000-0002-5278-2441], Avron Barr2, Robby Robson3, Xiangen Hu4, & Arthur Graesser4 1U.S. Army Research Laboratory, 2IEEE Learning Technologies Standards Committee, 3Eduworks, Inc., 4University of Memphis robert.a.sottilare.civ@mail.mil avron@aldo.com, robby.robson@eduworks.com, {art.graesser; xiangenhu} @gmail.com Abstract. This paper describes the purpose, goals, and guiding questions for the Adaptive Instructional System (AIS) standards workshop within the 2018 Intel- ligent Tutoring Systems (ITS) Conference Industry Track. Adaptive instructional systems (AISs) use human variability, learner goals and preferences, and other learner/team attributes along with instructional conditions to develop/select ap- propriate strategies (domain-independent policies) and tactics (actions). The goal of adaptive instruction is to optimize learning, performance, retention, and the transfer of skills between training environments and the work or operational en- vironment where the skills learned during training are to be applied. The Institute for Electrical and Electronics Engineers (IEEE) Learning Technologies Stand- ards Committee (LTSC) established a study group in December 2017 to evaluate the efficacy of AIS standards and the authors of this paper proposed this work- shop (and several others) to inform stakeholders and solicit their participation. The interaction with stakeholders at the ITS conference will be through the ideas presented in paper presentations and via an expert panel composed of the authors of this paper and other authors in this workshop. Keywords: Adaptive Instructional Systems (AISs), Intelligent Tutoring Sys- tems (ITSs), IEEE standards, Learning Technologies Standards Committee (LTSC) 1 Introduction Adaptive instructional systems (AIS) are defined as: “computer-based systems that guide learning experiences by tailoring instruction and recommendations based on the goals, needs, and preferences of each learner in the context of domain learning objec- tives” [1]. Examples of adaptive instructional systems include, but may not be limited to: intelligent tutoring systems (ITSs), intelligent mentors, recommender systems, per- sonal assistants for learning (PALs), and intelligent media (e.g., webpages) where the Back to Table of Contents 50 content, frequency, and interaction (e.g., support) provided is tailored to the needs, goals, and preferences of the learner or team of learners. In December 2017, the IEEE Learning Technologies Steering Committee (LTSC) formed a 6-month Standards Study Group to investigate the possible market need for standards across AISs. A recent AIS standards workshop in Orlando, Florida high- lighted several problems related to the authoring and maintenance of AISs that could be resolved by improving the interoperability of AIS components. This paper discusses the purpose, goals and guiding questions for AIS standards and is a companion docu- ment to a live panel of experts who will discuss progress and opportunities for AIS standards. 2 Workshop Purpose and Goals The purpose of this workshop is to educate AIS stakeholders about the IEEE study group and expose them to some of the proposed approaches to standardization through the paper presentations accepted for this workshop. A primary goal is to solicit stake- holder participation in the study group and any resulting IEEE working groups formed in the future in order to bring a diverse set of views and participation to bear in the standardization process. Several interactions with stakeholder communities point to broad interest in AIS standards. Discussions to date indicate opportunities to influence their affordability, their interoperability and reuse, making them more appealing learn- ing technology solutions for the masses. By the time of this presentation at the ITS conference, we will also have additional information to share about any proposals for working groups. Through an interactive process, the expert panel will provide insight on activities to standardize terms, elements and functions of AISs. The panel members will interact with ITS Conference audience to answer questions and receive feedback from AIS stakeholders. 3 Guiding Questions for AIS Standardization Several areas of AIS technologies are potential candidates for standardization. Robson, Sottilare & Barr [2] posed three essential questions to be considered during the IEEE standardization development process: • What do we want our standard(s) to do? • What do successful standards look like? • What is the appropriate process for developing standards? 3.1 What do we want our standard(s) to do? The primary answer to this question is “solve marketplace problems”. There must be a perceived value for any standard which is greater than the cost of implementing it. An- Back to Table of Contents 51 other driver is breakeven. Typically, business decisions (e.g., implement or don’t im- plement a standard) are based on the premise of breaking even within two years. If the cost of the standard is so high that it takes longer than two years to get to a breakeven point, it is likely that the standard will not be implemented. This is especially true if there are other ways to solve the problem (e.g., innovation). If the standard (e.g., in- teroperability, quality, convenience, or usability) solves a problem and does it with minimal cost or requires only a short amount of time to recover the investment, the standard will be perceived to have value. 3.2 What do successful standards look like? Given that we (as developers of standards) believe our standard has value, what is a reasonable measure of its success? Successful standards are adopted in relevant mar- kets and are sustained in those markets. Sometimes standards are adopted based on policy, but the policymakers may be short-sighted with respect to the long term value of the standard. The persistence of a standard in the marketplace is a sign of value and a measure of a successful standard. Other measures of a successful standard are the diversity of the community of inter- est who participate in the development of the standard and who ultimately adopt the standard. It is critical that the working group that develops the standard are fully rep- resentative of the global community who are intended to adopt the standard. Represen- tation from governments, industry members, and academics are one dimension of di- versity. Another dimension of diversity are the countries and regions of the global community. 3.3 What is the appropriate process for developing standards? The journey can often be as important as the destination when it comes to the standard- ization process. We advocate for standards development through an open, consensus- building process where all stakeholder groups are represented and active. This will give AIS standards the best possible chance of being successfully adopted, will aid in the growth and application of AIS technologies, enhance pathways for products to enter the marketplace and reduce the probability that AIS standards will restrict innovation and creativity in the AIS marketplace. 4 Discussion on Potential AIS Standards There are several AIS standards candidates that have been presented and discussed dur- ing the six months that the IEEE AIS standards study group has existed. Workshops like this one at ITS 2018 have engaged stakeholders and identified approaches ranging from conceptual models to component interoperability to common data structures for learner, domain, and instructional models. Below are four short descriptions of ideas that seem to be taking hold and have potential to meet the criteria discussed in Section 3 of this paper: Back to Table of Contents 52 • Common AIS Conceptual Model - a hierarchical common understanding of the com- position of AISs to aid engineers and scientists in communicating their designs and ideas in lectures, presentations, and technical papers as well as in system specifica- tions • AIS Component Interoperability and Reuse – a model of interoperability based on the common AIS conceptual model to facilitate integration and reuse of components through a set of common data messages • Common AIS Learner Model Features – a hierarchical common understanding of the most common features and their representation in both AIS short and long-term learner models • AIS Validation Standards – a testbed methodology to validate AIS compliance, in- teroperability, and compatibility with adopted AIS standards, and assess AIS system and component effectiveness Discussion within this workshop covers all of the potential AIS standards. Mr. Avron Barr, IEEE LTSC, led off the AIS standards workshop with a presentation cen- tering on the IEEE standards process and specifically addressed why we need standards. Mr. Barr also addressed the difficulties in the process and the characteristics of a good standard. Following this discussion, we presented and discussed papers covering stand- ardization ideas with respect to: • Component Interoperability (Sottilare & Brawner, p. 55) • Learner & Domain Models (McCoy, p. 63) • Pedagogy (McCoy, p. 63, DeFalco, p. 73) The focus of Sottilare & Brawner’s paper examined the use of the Generalized In- telligent Framework for Tutoring (GIFT) [3, 4] as a model for examining how compo- nent interoperability in AISs might work. In his presentation, Sottilare emphasized the importance of GIFT’s modular architecture in component reuse across a variety of GIFT-based tutors. McCoy made a pitch for common learner model and domain model structures along with item analysis and assessment standards. Finally, DeFalco put forward the idea of a metadata tagging schema based on a revision of Bloom’s cognitive taxonomy [5, 6]. All of these ideas resonated with the workshop participants and gen- erated significant discussion. 5 Next Steps A project authorization request (PAR) was submitted to IEEE LTSC in May 2018 and based on the plethora of activity in AIS stakeholder communities, we anticipated and received approval for an IEEE AIS Working Group in June 2018. Next steps will in- volve additional stakeholder recruiting to round out the diversity of the IEEE Project 2247 working group. Discussions about the organization, development of formal goals, and assignment of roles and responsibilities should quickly follow. We anticipate the Back to Table of Contents 53 development of a formal definition for AISs and a conceptual model for AISs that will guide future AIS standards development. Also high on the agenda is understanding of what is and is not included in the family of technologies known as AISs. This will help us examine appropriate exemplar sys- tems and discuss their attributes as part of standards discussions on conceptual models and component interoperability. Since data-driven models are a hot topic, we anticipate some discussion about the process of developing these types of models as a community to speed their development [7]. Acknowledgments A portion of the research described herein has been sponsored by the U.S. Army Re- search Laboratory. Statements and opinions expressed in this paper do not necessarily reflect the position or the policy of the United States Government, and no official en- dorsement should be inferred. References 1. Sottilare, R. & Brawner, K. (2018, March). Exploring Standardization Opportunities by Ex- amining Interaction between Common Adaptive Instructional System Components. In Pro- ceedings of the First Adaptive Instructional Systems (AIS) Standards Workshop, Orlando, Florida. 2. Robson, R., Sottilare, R. & Barr, A. (2018, June). Examining Barriers to the Adoption of IEEE Standards for Adaptive Instructional Systems (AISs). In the Exploring Opportunities to Standardize Adaptive Instructional Systems (AISs) Workshop of the 19th International Conference of Artificial Intelligence in Education (AIED), London, England, United King- dom, June 2018. 3. Sottilare, R.A., Brawner, K.W., Goldberg, B.S. & Holden, H.K. (2012). The Generalized Intelligent Framework for Tutoring (GIFT). Concept paper released as part of GIFT soft- ware documentation. Orlando, FL: US Army Research Laboratory – Human Research & Engineering Directorate (ARL-HRED). Retrieved from: https://gifttutoring.org/attach- ments/152/GIFTDescription_0.pdf 4. Sottilare, R., Brawner, K., Sinatra, A. & Johnston, J. (2017). An Updated Concept for a Generalized Intelligent Framework for Tutoring (GIFT). Orlando, FL: US Army Research Laboratory. May 2017. DOI: 10.13140/RG.2.2.12941.54244. 5. Bloom, B., Krathwohl, D. Taxonomy of Educational Objectives: The Classification of Ed- ucational Goals. Longmans, Green, New York, NY (1956). 6. Anderson, L., Krathwohl, D. R., et al (Eds.). A Taxonomy for Learning, Teaching, and As- sessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Allyn & Bacon, Bos- ton, MA (2001). 7. Sottilare, R. (2018, May). Community Models to Enhance Adaptive Instruction. In Foun- dations of Augmented Cognition (pp. 78-88). Springer International Publishing. Back to Table of Contents