Gaining Insight into Effective Teaching of AI Problem-Solving Through CSEDM: A Case Study Spencer Yoder Cansu Tatar Ifeoluwa Aderemi North Carolina State North Carolina State North Carolina State University University University smyoder@ncsu.edu ctatar@ncsu.edu iwaderem@ncsu.edu Sankalp Boorugu Shiyan Jiang Bita Akram North Carolina State North Carolina State North Carolina State University University University sboorug@ncsu.edu sjiang24@ncsu.edu bakram@ncsu.edu ABSTRACT Keywords CS education for all has been established as an important K-12 AI Education, Breadth-first-search AI algorithm CSEDM area of focus for both researchers and practitioners in re- for AI Education cent years. At the same time, due to the increasing preva- 1. INTRODUCTION lence of AI technology in humans’ everyday lives, artificial It has been almost half a decade since Simon Papert intro- intelligence (AI) education for K-12 is gaining special at- duced the idea of computer science (CS) for all [14]. Since tention among CS educators. AI literacy, even more than then, many researchers[15], educational institutes (e.g. [13]) general CS competencies, requires evidence-based research and organizations [18] have picked up the effort to create ac- to be effectively integrated in our schools. The common cessible CS curricula for K-12 classrooms. Meanwhile, artifi- learning environments utilized for CS education enable us cial intelligence, a sub-field of computer science, is becoming to go beyond conventional educational research approaches a domineering aspect of people’s everyday lives [17]. Daily by providing a platform where detailed data can be collected technologies and decisions are becoming more and more de- from students’ interaction with CS-education-related activ- pendent on AI. Under these circumstances, it is imperative ities. Thus, conventional educational research approaches for our new generation to gain a fundamental understand- coupled with insights gained from pattern recognition and ing of AI mechanisms and also its potential to introduce student modeling approaches enable us to effectively im- biases and unfairness through automated decision-making. prove our instruction and to provide students with adaptive Furthermore, the advances in AI systems are substituting scaffolding. In this work, we present our first AI curriculum many of the old jobs with ones that require the ability to do module that is designed to teach a fundamental AI search problem-solving with AI. For the aforementioned reasons, algorithm, Breadth-First Search (BFS), through a series of efforts for integrating AI into K-12 curriculum as an impor- progressively scaffolded activities. Data is collected from tant part of CS education is gaining momentum [11], [7]. a preliminary pilot of this activity with a high-school stu- dent in the form of a think-aloud protocol, screen capture, Integrating AI education into the K-12 curriculum poses sig- submitted block-based programming artifacts, and interview nificant challenges for educators since teachers often lack questions. Our results demonstrate that our activities have prior education in CS related fields [11]. Furthermore, the been successful in increasing the student ’s knowledge about skills, knowledge, and abilities required for getting engaged the BFS algorithm and more importantly, how this particu- in AI-related activities are novel to students and do not align lar AI algorithm can be utilized to solve real world problems. with their conventional ways of learning [cite]. Thus, effec- Based on the results of this pilot study, we propose design- tive instructional approaches should be identified through ing a comprehensive AI curriculum contextualized within a evidence-based research. This is especially important since learning environment that collects detailed data from stu- strong stereotypes and biases about who can learn this field dents’ progress to inform instructional design and facilitate prevent some of the students from getting engaged in the adaptive scaffolding for students. devised curriculum in the first place [12]. This can be facili- tated by taking advantage of one of the main affordances of CS learning environments that is their inherent capability to collect fine-grained data from students’ interactions with the learning activities. The collected log data can then be utilized in data-driven analysis that can inform instructors about the curriculum effectiveness and students’ learning. In this work we present the preliminary design of an AI cur- Copyright ©2021 for this paper by its authors. Use permitted under Cre- ricular module that teaches one of the most fundamental AI ative Commons License Attribution 4.0 International (CC BY 4.0) search algorithms, breadth-first search (BFS). The module Figure 1: West, the character to guide students through the exercises. consists of a series of activities that aims to teach learners about the BFS algorithm and ends with a coding activity where students have to incorporate the BFS algorithm in a real-world problem scenario to build a contact tracing ap- plication for COVID-19. While many devised AI curricula focus on the machine learning (ML) aspect of AI, we aim Figure 2: The graph representation of a city map. to provide K-12 students with a comprehensive view of AI that includes ML as an integrated aspect of AI. We present In this paper, we present a preliminary study that investi- a case study [1] where this curricular module is piloted with gates how students learn in a series of carefully-designed ac- a high-school student and data is collected through think tivities that introduce them to automated problem-solving aloud protocol [3], screen recording, and clinical interviews approaches and their ability to leverage AI techniques to [8]. This data is then analyzed to make inferences about the improve their decision-making process. effectiveness of the curriculum, required improvements and implications for design for a comprehensive AI curriculum integrated within a learning environment that collects learn- 3. ACTIVITY DESIGN ers’ interaction data to provide both educators and students We developed 5 exercises in UC Berkeley’s Snap block-based with adaptive assessment and feedback. programming language that introduce and use a simple graph- ical interface for teaching the breadth-first search algorithm Section 2 discusses related work. The details of the devised (BFS). The first four exercises serve as an introduction to curriculum are presented in section 3. The case study is BFS, and feature a character, West, pictured in Figure 1, presented in section 4, and the results of the pilot study are who guides student through the steps of the algorithm by discussed in section 5. Section 6 concludes the work, and telling them how to act as a GPS. The last activity is for- section 7 discusses implications for design and future work. matted as a programming exercise, where students are given an incomplete implementation of BFS, and are required to fill in the gaps. The algorithm is used for contact tracing, 2. RELATED WORKS similar to the COVID contact tracing apps, to showcase a Emerging research is exploring the design of learning ex- real-world application of BFS. periences to foster youths’ Artificial Intelligence (AI) liter- acy so that they are prepared to enter and engage with an AI-filled future [10]. In particular, efforts have been made 3.1 Exercise 1 to engage youth in understanding AI through the develop- Exercise 1 is designed to introduce graph search as a problem ment of machine learning (ML) models. For instance, a work in the context of a GPS navigating between cities, as well as by Zimmermann-Niefield and colleagues (2019) provided an to show how the methodologies for performing this search embodied learning experience for youth to create ML mod- differ between computers and humans. West shows the stu- els for recognizing their own physical activities [19]. They dent a city map as a graph, with nodes representing cities found that youth developed an understanding of how ML and transitions representing roads between them (Figure 2). models learned patterns of body movements and this could The student is then instructed to click through the cities to contribute to the understanding of the iterative process of find the shortest path from a starting city to a goal. The ML. In addition, Google developed web-based tools (e.g., student does this first with the simple map shown in figure Teachable Machine) to make ML accessible to the public, 2, then with a more complicated map, and then finally with including youth [3]. These studies stressed the cultivation a map which has cities hidden from the student, which can of data literacy among youth as modeling data as a core con- only be shown once they click an adjacent city. This last ex- cept in ML. While K-12 AI education is gaining momentum, ample is meant to describe the perspective of the algorithm the curricula being devised for K-12 students group tend to to the students, to contrast it with the human perspective focus on basic ML approaches. However, these curricula fail of seeing the whole state space at once. to demonstrate the connection between ML techniques and the broader schema of automated problem-solving [5, 9, 6]. 3.2 Exercise 2 As AI technology is being more thoroughly integrated into Exercise 2 is designed to introduce BFS as an algorithm for our everyday lives, it is imperative for students to become performing graph search. BFS is introduced to the students familiar with a holistic view of problem solving with AI. via a series of animations and they are instructed to perform was inspired by the COVID contact tracing apps that have recently been developed and implemented in order to show students a real-world application of BFS. Figure 4, right shows the graph of people with the infected people in red. Students were given a partial implementation of the algo- rithm along with missing blocks, with some instructions on how to snap the blocks into their proper place (Figure 4, left). Upon running the algorithm, if it is correctly imple- mented, students see the path from the user-selected person to the nearest infected person. 4. PRELIMINARY CASE STUDY 4.1 Research Design This study employed the case study method [16]. This method has been seen as a framework to determine the Figure 3: The graphical interface students interact with to problems that need to be studied in a bounded system [16]. perform BFS. (Bottom-left) The legend, which explains to Furthermore, Creswell and Poth [4] elaborate that the case students how the different states for nodes and transitions study approach is a research methodology that aims to iden- are displayed. (Middle) The graph representing a city map. tify specific cases and to explore in-depth content related to (Right) The fringe. these cases. Since this study aims to understand problems, challenges, and opportunities while piloting the AI curricu- lum module within a bounded system, this methodology was each of the steps of BFS by interacting with our graphical in- followed for the research design. terface, shown in Figure 3. This interface provides two ways to perform the actions of the algorithm. The first is to add nodes to the fringe by clicking them on the displayed graph. 4.2 Participant and Context The second is to remove nodes from the fringe. The leg- The case study participant was Selim (pseudonym). He was end in Figure 3, which shows the graphical representations a student in the 11th grade at a public high school in the of each state for the nodes and transitions, is displayed and United States. He had an interest in Artificial Intelligence the interface updates accordingly as students interact with and was actively looking for opportunities to learn about it. how AI worked and how AI could be used in the field of his interest, Biomedical Engineering. He had some general knowledge about AI and Python programming. The study 3.3 Exercise 3 took place in the context of two informal virtual sessions Exercise 3 consists of a completely guided run-through of that were designed to introduce Artificial Intelligence with BFS for a medium-sized graph with eight nodes and ten the BFS search algorithm. The sessions were conducted re- transitions. At every step of the algorithm, students are motely, a week apart, via Zoom, a video conferencing tool. told what to do next and upon making a mistake, they are The first session took approximately one hour. In this ses- informed about what the mistake is and re-prompted until sion, we first described the goal of the session and empha- choosing the correct option. sized that the student was expected to experience the first four learning activities as a user, complete a post-activity 3.4 Exercise 4 questionnaire, and participate in an interview. The second Exercise 4 is composed of a partially guided run-through session was mainly focused on the fifth activity and took 1.5 of BFS. To connect this exercise with a real-world applica- hours. In this session, Selim answered questions related to tion, we show the graph superimposed over a map of the the previous session, followed a written Snap tutorial, com- east coast of the U.S., with the nodes named for their cor- pleted the activity, and participated in the interview. responding cities. For the duration of the exercise, students complete the steps of the algorithm without feedback unless 4.3 Data Collection Process they make a mistake. Upon making a mistake, they are in- In this study, we collected data in the form of a think- formed about the nature of their mistake and re-prompted aloud protocol with approximately thirty minutes of a semi- until they perform the correct next step. structured interview, post-questionnaire, screen capture, and submitted block-based programming artifacts. In the first 3.5 Exercise 5 session, we asked the participant to answer warm-up ques- Exercise 5 challenges students to implement their under- tions (e.g., Have you ever taken any class on AI?) and com- standing of BFS as Snap code. For this exercise, we re- plete an activity that aimed to extract his interests and contextualized the state space for BFS from a map of cities general knowledge about AI. In addition to the warm-up to a social network of people. When the exercise is initial- activity, at the end of each learning activity, we also asked ized, one to three random people are marked as infected “What do you think you have learned? What else would you with COVID-19. The objective for BFS was changed from like to explore?”. With these questions, the participant re- finding the shortest path from a starting city to a goal city flected upon his own learning. Also, after he completed the to finding the shortest number of social connections from activities, we asked him to complete the post-questionnaire a user-selected person to an infected person. This scenario that included seven multiple-choice questions related to the Figure 4: Exercise 5. (Left) The incomplete BFS implementation with the missing blocks located under the code. (Right) the graph representing the social network from which the student selects an arbitrary person to trace to the nearest infected person. BFS learning activities. At the end of the first session, we an interdisciplinary field. Selim was interested in the field conducted semi-structured interviews for approximately 10 of biomedical engineering and believed that AI would con- minutes to learn more about his experience. In the second tribute to innovations in every field, including biomedical session, we followed a similar structure. We started the ses- engineering. Thus, he was motivated to learn more about sion with the warm-up questions to recall the student’s un- the application of AI in the field of his interest. During derstanding from the previous session and to correct miscon- the first session, he shared, “If you are trying to determine, ceptions. Before the activity, we guided the student through for tissue engineering, maybe you could use a model to de- the creation of pseudocode which reflected his understanding termine what’s the best material to use based on different of the algorithm. We then provided a written Snap tutorial situations.” At the end of the first session, Selim asked the to allow the student to get familiar with the learning envi- research team about learning experiences in biomedical en- ronment. After he skimmed through the tutorial document, gineering and computer science in higher education since he moved on to the fifth activity. At the end of the ac- he was at the critical stage of choosing future careers. His tivity, we conducted an interview to learn more about his question highlighted the need of supporting students to de- experiences. velop interdisciplinary learning and collaboration skills and understand the integration of AI in fields of their interest. 4.4 Data Analysis To analyze the transcripts of interviews, we followed the 5.2 Creating opportunities for students to dis- thematic coding method [2], which is a widely used method to analyze qualitative data for identifying, organizing, and cuss who benefits from AI and is left out describing themes. In the first round of the analysis, one re- Our analysis indicates that topics related to AI ethics might searcher analyzed the data to explore learning challenges serve as a catalyst for students to have in-depth discussions and opportunities. After the first-round coding, a peer- about who benefits from AI and who is left out. For instance, debriefing meeting was held with three other researchers in the excerpt below (Excerpt 1), guided by the research to ensure the trustworthiness of the findings. Addition- team, Selim emphasized that AI ethics could be an engaging ally, we analyzed the student’s process of building the block- topic of discussion. Excerpt 1. based programming artifacts by following the thematic cod- ing strategies. We open-coded the students’ actions in the 1. Research team: Have you talked about AI with your activities to identify learning challenges and opportunities. family and friends? After that, we discussed these codes and came up with the general themes discussed in section 5.3. 2. Selim: Not really my friends. With family, yeah. We’ve had a few discussions. Um, it’s probably a really com- 5. RESULTS AND DISCUSSION mon discussion. I think one of my parents heard it Results are organized around three overarching themes: teach- on a podcast or something like the thing about a self- ing AI as an interdisciplinary field, creating opportunities driving car having an unavoidable accident. for students to discuss who benefits from AI and is left out, and promoting effective AI learning through developing a 3. Research team: Yeah, I remember this one. humanized curriculum. 4. Selim: Like, who it (referring to the self-driving car) 5.1 Teaching AI as an interdisciplinary field picked to crash into or something. Just like ethics of AI We should provide opportunities for students to understand is probably the most interesting conversation to have the integration of AI in application areas and teach AI as with other people. In Turn 4, he highlighted that a model would fall short in ad- student was unclear about how to map from steps of his dressing the moral question of who a self-driving car should pseudocode onto the different sections of the partial code. kill in an unavoidable accident. However, he did not rec- ognize that essentially, it’s the car maker or designer who While the first challenge can be addressed by providing stu- would make the decision. dents with a more extensive block-based programming tuto- rial beforehand, the second two challenges require improve- ments in the design of the activity. For example, a better de- 1. Research team:Do you find AI ethics interesting? scription of the functionality of custom blocks and a clearer distinction between custom blocks and contextually named 2. Selim: Oh, yeah. variables can alleviate some of the confusions that students 3. Research team: In what ways? Can you give an exam- might encounter when analyzing the partial code presented ple? to them. Additionally, having students implement a simple case of BFS following the same context as the previous ac- 4. Selim: So another thing that I found interesting is the tivities might facilitate the transition to implementing the hiring process. I think it’s Amazon. It has implicit BFS algorithm in an arbitrary real-world context. bias against hiring applicants that had women in their application. That’s obviously an important issue that 6. CONCLUSION needs to be fixed. So there are really complex issues With the advances of AI technology and its rapid integra- when you talk about AI ethics. tion within our society, it is imperative that AI education is included as part of the K-12 standard curriculum. In In turn 8, he raised the concern about gender bias using AI this study, we presented the preliminary design of a cur- recruiting or resume screening tools. Overall, we can see that ricular module that is designed to teach a foundational AI AI ethics and AI decision making that would have an impact algorithm, breadth-first search (BFS), to high school stu- on who benefits and is left is a promising topic for students dents. We presented the results of a pilot implementation of to think critically about the impacts of AI technologies. this curriculum with a high-school student as a case study. Our results demonstrated that this intervention helped the student gain a better perception about the interdisciplinary 5.3 Promoting effective AI learning through nature of AI and how it can be incorporated to enrich other developing a humanized curriculum fields. Furthermore, the discussions fostered curiosity around As described in section 3, the student is presented with a AI ethics which is an important aspect of AI education. Fi- series of five activities. The first four activities are designed nally, our analysis of the student’s interactions with the de- to introduce the BFS algorithm to the student while the fifth signed activities and his reflections showed that the activities requires him to implement the BFS algorithm in a block- were successful in improving the student’s understanding of based programming environment in the context of a contact the BFS algorithm while posing some challenges when the tracing app. Selim progressed though the first four activities student tried to map from the algorithm to a block-based rather smoothly. After each activity he indicated a deeper programming implementation. understanding of the BFS algorithm. He then successfully answered content knowledge questions presented to him at 7. FUTURE WORK the end of the first session In the future, we aim to utilize insights obtained from this case study to not only improve the design of the current In the beginning of the second session, the student started by module but also to inform design of the future AI-related reflecting on what he recalled from the previous session and curricular modules. While this study provided a qualitative was then asked to devise pseudocode for the BFS algorithm. analysis of observed interactions between the student and Through this process he demonstrated a good understanding activities, we plan to automate this process in the future of the BFS algorithm but a lack of familiarity with writing through data-driven approaches to obtain more insights into pseudocode. With the guidance of researchers he managed the effectiveness of our curriculum and students’ learning. to turn his knowledge of the BFS algorithm into a pseu- We propose creating a series of curricular modules that teach docode format. He was then presented with an incomplete students a holistic view of AI and equip them with knowl- contact tracing app that utilizes BFS to find the shortest edge, skills, and abilities that prepare them for conducting path between an arbitrary person and an infected person problem-solving with AI in humanized real-world scenarios. in his social circle.The BFS algorithm was partially imple- We further propose situating our curriculum within a learn- mented as a Parson’s problem, and the student snapped the ing environment where data can be collected and analyzed readily available blocks on the screen to complete the al- for informing instruction and providing students with adap- gorithm. Here again, the student showed an adequate un- tive scaffolding. derstanding of the BFS algorithm. However, he faced some challenges when trying to implement it within the block- based programming environment. We identified three main 8. REFERENCES themes of challenges faced by the student while conducting [1] M. Bassey. Case study research. Research methods in the programming. First, he was not familiar with some of educational leadership and management, pages the blocks, list-related blocks in particular, and was con- 108–121, 2002. fused about what they represent in the incomplete program. [2] V. Braun and V. Clarke. Using thematic analysis in Secondly, he had trouble understanding the functionality of psychology. Qualitative research in psychology, some of the custom blocks provided for him. Finally, the 3(2):77–101, 2006. [3] E. Charters. The use of think-aloud methods in S. K. Kane, and R. B. Shapiro. Youth learning qualitative research an introduction to think-aloud machine learning through building models of athletic methods. Brock Education: A Journal of Educational moves. In Proceedings of the 18th ACM International Research and Practice, 12(2), 2003. Conference on Interaction Design and Children, pages [4] J. W. Creswell and C. N. Poth. Qualitative inquiry 121–132, 2019. and research method: Choosing among five approaches, 2007. [5] J. Estevez, G. Garate, and M. Graña. Gentle introduction to artificial intelligence for high-school students using scratch. IEEE Access, 7:179027–179036, 2019. [6] T. Hitron, Y. Orlev, I. Wald, A. Shamir, H. Erel, and O. Zuckerman. Can children understand machine learning concepts? the effect of uncovering black boxes. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pages 1–11, 2019. [7] S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, and R. Luckin. Artificial Intelligence in Education. Springer, 2019. [8] V. R. Lee and D. A. Fields. A clinical interview for assessing student learning in a university-level craft technology course. 2013. [9] P. Lin, J. Van Brummelen, G. Lukin, R. Williams, and C. Breazeal. Zhorai: Designing a conversational agent for children to explore machine learning concepts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13381–13388, 2020. [10] D. Long and B. Magerko. What is ai literacy? competencies and design considerations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–16, 2020. [11] P. S. Mohammed et al. Towards inclusive education in the age of artificial intelligence: Perspectives, challenges, and opportunities. In Artificial intelligence and inclusive education, pages 17–37. Springer, 2019. [12] D. J. Moran. The need for evidence-based educational methods. In Evidence-based educational methods, pages 3–7. Elsevier, 2004. [13] S. Nikiforos, C. Kontomaris, and K. Chorianopoulos. Mit scratch: A powerful tool for improving teaching of programming. Conferrence on Informatics in Education, pages 1–5, 2013. [14] S. Papert. The children’s machine: Rethinking school in the age of the computer. ERIC, 1993. [15] D. Passey. Computer science (cs) in the compulsory education curriculum: Implications for future research. Education and Information Technologies, 22(2):421–443, 2017. [16] R. E. Stake. Qualitative case studies. 2005. [17] C. Tatar, M. Yoder, M. Coven, K. Wiedemann, J. Chao, W. Finzer, S. Jiang, and C. Rosé. Modeling unstructured data: Teachers as learners and designers of technology-enhanced artificial intelligence curriculum. In International Society of the Learning Sciences Annual Meeting 2021, 2021. [18] A. Tucker, F. Deek, J. Jones, D. McCowan, C. Stephenson, and A. Verno. A Model Curriculum for K–12 Computer Science. ACM/Association for Computing Machinery, 2003. [19] A. Zimmermann-Niefield, M. Turner, B. Murphy,