Fostering AI Literacy: An Investigation of Learning Resources for Secondary School Students Stephan Keller∗ , Gerald Steinbauer-Wagner Graz University of Technology Abstract Artificial Intelligence (AI) continues to transform industries and everyday life. It is increasingly important to foster AI literacy, particularly in secondary education. AI literacy equips students with the knowledge and skills to understand, interact with, and critically evaluate AI technologies. It empowers them to become informed citizens and active contributors in an AI-driven future. Many universities, organizations, and initiatives have created AI literacy learning resources and made them publicly available. However, to the best of our knowledge, there is no existing effort to categorize the learning objectives of these resources. This exploratory research uses a methodological approach to analyze and provide insights into the growing landscape of AI literacy resources available for secondary education. We analyzed selected resources in-depth to provide a better understanding. Therefore, we identify and assign individual learning objectives according to competence dimension and literacy type. We measure the broadness and diversity along these dimensions through this structured approach. This result can help refine resources to cover AI literacy holistically. With this approach, we can compare AI literacy resources and perhaps contribute to future improvements in AI literacy resources. Keywords AI literacy, Secondary education, Learning resources 1. Introduction AI literacy has emerged as a critical competency in the educational landscape, reflecting the pervasive integration of AI technologies into various aspects of daily life. The concept of AI literacy encompasses a range of skills, including understanding AI principles, ethical considerations, and the ability to engage with AI systems critically [1, 2]. This multifaceted nature of AI literacy is underscored by recent systematic literature reviews, which highlight the necessity for educational frameworks that equip learners with the knowledge and skills to navigate an AI-driven world effectively [3, 4, 5, 6]. Touretzky’s AI4K12 initiative is a foundational reference in this domain, supporting the inclusion of AI education in K-12 curricula [2]. This initiative emphasizes the importance of fostering a comprehensive understanding of AI among students, preparing them for future challenges and opportunities presented by AI technologies [7]. The AI4K12 framework outlines essential learning objectives that cover technical aspects of AI and address ethical implications and societal impacts, thereby promoting a holistic approach to AI literacy. Moreover, recent literature reveals that effective AI education must integrate psychological compe- tencies alongside traditional knowledge frameworks. This integration is crucial for enhancing students’ self-efficacy in problem-solving and emotional regulation when interacting with AI systems [3, 8]. Research also indicates that fostering AI literacy can significantly influence students’ attitudes towards technology, encouraging them to engage with AI in a responsible and informed manner [9, 4]. AI literacy equips individuals with the tools to critically assess and utilize AI technologies in their personal and professional lives. As AI continues to evolve and influence various sectors, educational initiatives like AI4K12 are essential for ensuring that future generations are not only consumers of AI but 1st Workshop on Education for Artificial Intelligence (edu4AI 2024, https:// edu4ai.di.unito.it/ ), co-located with the 23rd International Conference of the Italian Association for Artificial Intelligence (AIxIA 2024). 26-28 November 2024, Bolzano, Italy ∗ Corresponding author. Envelope-Open stephan.keller@tugraz.at (S. Keller); steinbauer@ist.tugraz.at (G. Steinbauer-Wagner) Orcid 0000-0001-8258-4553 (S. Keller); 0000-0001-9374-7864 (G. Steinbauer-Wagner) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings also informed citizens capable of contributing to discussions about its ethical and societal implications [3, 7, 4]. The landscape of AI literacy resources is growing rapidly. The type of providers differ from commercial providers, universities, and educational research initiatives up to governmental interest in fostering AI literacy amongst citizens. The individual resources are, therefore, often not accompanied by scientific publications. Even if that is the case, the contents might have changed since the publications. For this reason, we decided on an unconventional approach to identify, collect, and screen current online resources on AI literacy. The global increase in available AI literacy resources leads us to the following research questions: RQ1 How diverse are the individual learning resources? How are the competence dimensions (Attitudes, Behavior, Cognition) distributed over all learning objectives? RQ2 How well is AI ethics literacy covered? 2. Method First, preliminary research is conducted to find related work and popular AI literacy frameworks and identify relevant keywords for further steps. Then, a systematic approach is used to identify resources, which are screened and analyzed. Selected resources are then analyzed in-depth, i.e. individual learning objectives are identified. Finally, we categorize these learning objectives into a competence dimension (Cognition, Behavior, Attitudes) and an AI literacy type (Generic, Ethic). This allows a comparison based on the holistic AI literacy assessment matrix [10]. 2.1. Identification We search for current AI literacy frameworks or curated materials, which are sometimes but not always accompanied by scientific literature. Therefore, we identify potentially relevant resources through a Google search. The incognito mode helps to avoid search history bias through the browser cache. We determine relevant search terms through preliminary research. Then, we combine those search terms as boolean logic search requests to optimize the search results and make it as reproducible as possible. ("term1" OR "term2" OR "term3") AND ("term4" OR "term5") All results on the first page, except sponsored results, are added to the list of identified targets. 2.2. Screening & Selection process The identified resources are then selected in a two-step screening process. In the 1st screening step, the type of target is analyzed. Since we are searching for high-quality learning resources appropriate for secondary school students, we exclude anything else. In the second screening, we look at diverse criteria: 1. Who published the resource? 2. Accessibility 3. Provided language(s) 4. Type of resource (mooc, videos, textual, plugged, unplugged...) 5. Appropriate for secondary school students 6. Quality of content 2.3. Analysis of selected resources Efforts to conceptualize AI literacy and assessment seem very specific and isolated. The holistic AI literacy assessment matrix [10] provides a structured and comprehensive way to assess AI literacy, ensuring that technical knowledge, practical application, and ethical considerations are accounted for. It addresses three core areas of AI literacy: Generic AI Literacy, Domain-Specific AI Literacy, AI Ethics Literacy. The matrix assesses AI literacy along three dimensions: Cognition, Behavior (skills), and Attitudes (values) following the “concept of competence” [11]. We use the matrix to categorize individual learning objectives of a selected resource along these dimensions and literacy types. Since, in our case, only educational AI literacy resources are selected, the domain-specific area becomes less relevant. We augment the original authors’ concept to inspect the holistic coverage of AI literacy resources. The individual classification depends significantly on the context, how the learning objective is defined, and how it is meant to be achieved. A practical exercise to achieve a learning objective is likely classified as Behavior competence. On the other hand, a purely theoretical component, where facts must be understood or memorized, is likely classified as Cognition competence. Attitude learning objectives are components that prompt self-reflection or require expressing one’s attitude toward a specific topic. 3. Results We identified currently available online learning resources on AI literacy, selected high-quality repre- sentatives, and inspected the diversity of the learning objectives. 3.1. Identification and screening Based on insights from the preliminary research, we formulated key search terms. These were combined into the nine search queries (see Table 1) to reach a representative number of results. Table 1 Boolean search queries used to identify potentially relevant targets 1 “AI Literacy material” 2 “AI literacy material secondary education” 3 “AI Literacy learning platform” 4 “AI Literacy” AND (“material” OR “material secondary education” OR “learning platform”) 5 (“Artificial Intelligence” OR “AI”) AND “Literacy” AND (“material” OR “material secondary education” OR “learning platform”) 6 (“Künstliche Intelligenz” OR “Artificial Intelligence” OR “AI”) AND “Literacy” AND (“material” OR “material secondary education” OR “learning platform”) 7 (“Künstliche Intelligenz” OR “KI” OR “Artificial Intelligence” OR “AI”) AND “Literacy” AND (“material” OR “resource” OR “course” OR “MOOC”) AND (“secondary education” OR “learning platform”) 8 “free online course” AND (“AI” OR “Artificial intelligence”) 9 “Artificial intelligence” AND “awareness” AND “material” The incognito search on these queries resulted in 85 targets (see Figure 1). From these 85 targets, 20 exact duplicates were removed immediately. In the first screening, we selected high-quality learning resources with a dedicated focus on AI literacy. During the process, 51 targets were excluded; 5 were excluded as duplicates since they refer to already identified targets or are simply in the same domain. 14 targets led to scientific publications, possibly relevant for further literature research, but didn’t contain actual material. 22 were blog-like articles on AI literacy and/or lists of external AI literacy-related resources. 8 sources were excluded because they were restricted to a paying audience or enrolled students at a specific institution. Two didn’t fit into our topic. Identification Resources identified from: Resources excluded before web search (n = 85 ) screening (n = 20) 1st Screening Resources screened (n = 65) Resources excluded (n = 51) 2nd Screening Resources screened (n = 14) Analysis Resources classified (n = 3) Figure 1: Resource selection The second screening investigated the remaining 14 targets (see Table 2) in more detail. We examined who published the resource and their motivations for doing so. Additionally, we verified if registration and or a fee is required to access or use the material. Further, we looked into the individual materials and investigated usability, optional languages, and age-appropriateness. Finally, we selected three targets that fulfilled all our criteria, offered good usability, and clearly stated the learning objectives. Table 2 Results of the 1st screening, the asterisk (*) marks the selected resources in the 2nd screening Name Type of resource CRAFT* collection of resources for high school teachers MIT RAISE collection of resources Common Sense Ed collection of lessons AI4Europe platform (AI4Europe project ) AI for Education course, resources, lessons Microsoft AI collection of resources Data AI Literacy collection of resources Figshare AI Lit Teaching material for AI literacy Motimore online course on AI media literacy KI-Campus learning platform AI4T platform, mooc Elements of AI* AI educational resources Ethics of AI MOOC course ENARIS* AI educational resources 3.2. Selected resources The three selected resources from the second screening were analyzed in-depth. In the first step, individual learning objectives were identified. These learning objectives were assigned to the holistic AI literacy assessment matrix [10]. The learning objectives distribution is shown in Table 3 and the competence distribution is shown in Figure 2. 3.2.1. Elements of AI Elements of AI is a free online course series that aims to make artificial intelligence more accessible to a wide audience. It was created by the University of Helsinki in collaboration with MinnaLearn, and was first launched in 2018. In 2021, the course was made publicly available in all official EU languages. The course is designed for people with no prior knowledge of AI or programming, although there is an advanced course available for those looking to deepen their technical skills. The primary objective of Elements of AI is to educate people about the fundamentals of AI and encourage informed discussions about the potential of AI in society. It seeks to demystify AI and make it approachable by combining theory with practical exercises. The course is part of a broader initiative to increase AI literacy and equip individuals with the skills and knowledge needed to understand the impact of AI on various industries and everyday life. The course offers a structured process and maps personal progress through exercises after each section. Upon completion, a certificate is available for purchase. With a total of 27 learning objectives, two-thirds were classified as Cognition. One-third of the learning objectives were classified as Behavior, and none as Attitudes. The number of learning objectives classified as Ethic in the two Elements of AI courses is surprisingly low (n=3). However, investigation during the 2nd screening revealed that the University of Helsinki also created a separate MOOC Ethics of AI. This might explain the lack of ethics-related learning objectives in Elements of AI. 3.2.2. ENARIS The international AI education and awareness project ENARIS [12] aims to spark interest in Artificial Intelligence in young students aged 10 to 14 and to foster their technical knowledge in a playful way. The material of ENARIS is created with a classroom environment in mind and can be used by interested teachers without limitations. The website offers 10 ready-to-use workshop-sized modules on different AI literacy topics in English, German, and Hungarian languages. Enaris has a total of 67 identified learning objectives, with 39 classified as Cognition, 20 as Behavior, and eight as Attitudes. In total 12 learning objectives were classified as ethic literacy, evenly distributed along the competence dimensions. 3.2.3. CRAFT The third selected target, CRAFT AI literacy resources, is a collection of free, high-quality AI literacy resources by Stanford University. These resources are “informed by learning sciences research for non-profit use by classroom educators”. The materials provided in the individual lessons are ready-to- use and specifically designed for high-school teachers. From 23 learning objectives in total, 13 were classified as Cognition, eight as Behavior, and three as Attitudes. Six learning objectives were classified as ethic literacy, evenly distributed along the three competencies. 4. Discussion This study highlights the critical role of high-quality AI literacy resources in secondary education, em- phasizing their importance in equipping students with both technical knowledge and ethical awareness. As AI technologies increasingly influence daily life, preparing students to engage with AI critically is essential. The analysis of available AI literacy resources demonstrates a growing variety of tools and platforms, each aimed at different aspects of AI education, from basic concepts to more advanced Table 3 Classification of learning objectives in competences and literacy type AI literacy type Generic Ethics Generic Ethics Generic Ethics Cognition 16 2 11 2 35 4 Behavior 8 1 6 2 16 4 Attitudes 0 0 1 2 4 4 Elements of AI Craft Enaris Figure 2: Competence dimensions distribution of learning objectives applications. However, gaps in accessibility, depth of content, and integration of ethical considerations remain challenges that must be addressed. The analysis of the three selected resources has shown a similar distribution of the learning objectives on the holistic assessment matrix. The classification of these high-quality resources could form the base for a benchmark for other AI literacy resources. Further research is necessary to identify meaningful metrics for a better comparison. Nevertheless, this study has certain limitations. While it provides insights into currently available resources, more research is needed to evaluate the effectiveness of these resources in diverse educational contexts. The proposed process to identify learning objectives and classify them into the holistic AI literacy matrix is yet to be improved. A semiautomatic process (e.g., through NLP) could support a consistent, measurable way to compare AI literacy learning resources. Another interesting metric to explore would be the number of learning objectives along the investigated competence dimensions and literacy types. Future studies could further explore how AI literacy programs impact student learning, particularly in underrepresented or underserved communities, and investigate the best methods for integrating ethical discussions into technical AI education. References [1] D. Long, B. Magerko, What is ai literacy? competencies and design considerations, Conference on Human Factors in Computing Systems - Proceedings (2020). URL: https://www.researchgate.net/ publication/339272039_What_is_AI_Literacy_Competencies_and_Design_Considerations. doi:10. 1145/3313831.3376727 . [2] D. Touretzky, C. Gardner-McCune, F. Martin, D. Seehorn, Envisioning ai for k-12: What should every child know about ai?, Proceedings of the AAAI Conference on Artificial Intelligence 33 (2019) 9795–9799. URL: https://ojs.aaai.org/index.php/AAAI/article/view/5053. doi:10.1609/AAAI. V33I01.33019795 . [3] G. Biagini, Assessing the assessments: Toward a multidimensional approach to ai literacy, Media Education (2024). doi:10.36253/me- 15831 . [4] D. D. Figaredo, Responsible ai literacy: A stakeholder-first approach, Big Data & Society (2023). doi:10.1177/20539517231219958 . [5] L. Casal-Otero, A. Catala, C. Fernández-Morante, M. Taboada, B. Cebreiro, S. Barro, Ai literacy in k-12: a systematic literature review, International Journal of STEM Education 2023 10:1 10 (2023) 1–17. URL: https://stemeducationjournal.springeropen.com/articles/10.1186/s40594-023-00418-7. doi:10.1186/S40594- 023- 00418- 7 . [6] D. T. K. Ng, J. K. L. Leung, S. K. W. Chu, M. S. Qiao, Conceptualizing ai literacy: An exploratory review, Computers and Education: Artificial Intelligence 2 (2021) 100041. doi:10.1016/J.CAEAI. 2021.100041 . [7] K. Kahn, N. Winters, Constructionism and ai: A history and possible futures, British Journal of Educational Technology (2021). doi:10.1111/bjet.13088 . [8] Y. Shen, Perceived support and ai literacy: The mediating role of psychological needs satisfaction, Frontiers in Psychology (2024). doi:10.3389/fpsyg.2024.1415248 . [9] H. A. Modran, Using the theoretical-experiential binomial for educating ai-literate students, Sustainability (2024). doi:10.3390/su16104068 . [10] N. Knoth, M. Decker, M. C. Laupichler, M. Pinski, N. Buchholtz, K. Bata, B. Schultz, Developing a holistic ai literacy assessment matrix – bridging generic, domain-specific, and ethical competencies, Computers and Education Open 6 (2024) 100177. doi:10.1016/J.CAEO.2024.100177 . [11] F. E. Weinert, Concept of competence: A conceptual clarification, in: D. S. Rychen, L. H. Salganik (Eds.), Defining and selecting key competencies, Hogrefe & Huber, Seattle, WA, 2001, pp. 45–65. [12] M. Kandlhofer, P. Weixelbraun, M. Menzinger, G. Steinbauer-Wagner, Ágoston Kemenesi, Education and awareness for artificial intelligence, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 14296 LNCS (2023) 3–12. doi:10.1007/978- 3- 031- 44900- 0_1 .