Exploratory Process Analysis of Teacher Learning of AI Integration through Collaborative Design Xiaoyu Wan⇤ Jingwan Tang⇤ Xiaofei Zhou⇤ University of Rochester University of Rochester University of Rochester xwan3@u.rochester.edu jtang21@u.rochester.edu xzhou50@ur.rochester.edu Zhen Bai University of Rochester zbai7@ur.rochester.edu ABSTRACT education programs for AI integration. Little research has been done on the study of computer- supported collaborative learning (CSCL) in the context of Our study attempts to address this inquiry by investigat- teacher learning, especially the temporal analysis of the knowl- ing teachers’ learning process using existing data collected edge construction process and its impact on learning out- from a professional development program, ML4STEM, in comes. The purpose of our research is to explore multi- April 2020 [59]. It introduced an ML-enhanced scientific dis- ple temporal analysis methods to understand the knowledge covery learning environment to 18 in-service K-12 teachers construction in K-12 teacher CSCL of ML-empowered les- and engaged them in several learning activities to learn to son plan design using the video transcript data. The social teach with a new tool. This paper specifically focuses on the network analysis yielded high and low meta-cognition across collaborative design activity in a computer-supported col- groups and indicated the association with the design arte- laborative learning context (CSCL). Teachers created ML- fact quality. Sankey diagram visualization demonstrated the enhanced lesson plans (Fig. 7) facilitated by a web-based macro-level cognition activity flow in the process. Lag se- learning environment, SmileyDiscovery (Fig. 6), enabling quential analysis found patterns of transition of technolog- novice learners to apply k-means clustering in science con- ical, pedagogical, and content knowledge during collabora- text to discover patterns and new knowledge [59]. tive design contributing to the group learning outcome. A discussion on the results is provided, which sheds light on an- Collaborative design has been argued as the most e↵ective alyzing and facilitating teacher learning in CSCL settings. method to support teachers’ understanding of technology integration in classrooms [24]. However, the desired learn- Keywords ing outcomes are not naturally guaranteed [26]. An e↵ec- Computer-supported collaborative learning (CSCL), tempo- tive CSCL learning process depends on constructing new ral analysis, teacher learning, knowledge construction knowledge and generating new understandings during the collaboration process [10, 15]. Particularly, d [21]. To un- derstand the quality of such knowledge construction process, 1. INTRODUCTION a key to di↵erentiating the quality of cognitive activities at Artificial intelligence (AI) plays an increasingly critical role high and low levels becomes necessary, as we expected the in K-12 education as technology advancement [19]. It im- desired high level of knowledge construction during the col- poses a new requirement for K-12 teachers with limited com- laboration [43]. Previous research related to CSCL lacks puting backgrounds to develop an understanding of teaching such e↵orts in studying the knowledge construction process with AI technologies in classrooms [13, 32]. Recent research in the teacher learning context [26] or limited to descriptive e↵orts started initiating professional development programs analysis to reveal the factors within CSCL that contribute to prepare teachers with sufficient knowledge about utilizing to teacher learning [18, 34, 4, 27]. AI to support student learning with subject matters [47, 52, 58]. These studies, however, provide little information about To uncover patterns of teachers’ collaborative learning and how teachers engaged in sense-making activities of AI tech- investigate the knowledge construction process demonstrated nologies, which are essential in guiding the design of teacher by the quality of cognitive processing, we explored multiple ∗ These three authors contributed equally. methods to analyze the temporal data at both individual and group levels. Social network analysis is to uncover both the interactions between participants and with cognitive activi- ties to identify group collaboration patterns and suggest col- laboration strategies based on groups’ end-product of learn- ing [21]. Lag sequential analysis (LSA) is to investigate how knowledge is transiting between groups during collaborative design. Sankey diagram visualizes macro-level cognitive ac- tivity flow through the collaborative design process. The Copyright ©2021 for this paper by its authors. Use permitted under Cre- results provide insights into how di↵erent collaboration pat- ative Commons License Attribution 4.0 International (CC BY 4.0) terns across teams a↵ect learning and how knowledge con- structs across groups. The implications of such findings are shared learning activities for knowledge construction [20]. discussed at the end of this paper. However, such a shared process does not necessarily lead to productive knowledge outcomes. A Previous study showed 2. RELATED WORK the interrelations between cognitive events under knowledge building discourse and uncovered the sequential pattern of 2.1 Artificial Intelligence in K-12 Teaching events using frequent pattern mining and latent sequential AI technologies have become increasingly crucial in educa- analysis [7]. In addition, another work studied how low- tion by playing four roles: intelligent tutor, intelligent tutee, performing and high-performing groups progress through a intelligent learning tool & partners, and the policy-making framework of socially shared regulation of learning and ar- advisor [19]. Implementing them in classrooms, however, is gue the importance of recognizing challenges and strategies a challenge for K-12 teachers. One of the primary obstacles in group collaboration [31]. These two studies provided in- is that guiding students to learn with AI tools require teach- sights for uncovering the patterns of knowledge construction ers to understand relevant technological knowledge [19]. It in the teacher learning context. is another challenge to prepare K-12 teachers for teaching with AI in classrooms due to their limited computing back- Various methods have been used to study the process of grounds [13], and the lack of teaching materials [32]. To pro- CSCL, mainly in inferential statistics and the coding-and- vide insights on potential solutions to the aforementioned count approach [26, 26], including social network analysis challenges, we analyzed the learning process showing how analyzing the group interactions over time (e.g., [51, 50, 28]), K-12 STEM teachers learned collaboratively to design ML- sequential analysis studying the learning event patterns (e.g, empowered lesson plans. [9, 57]), and di↵erent types of visualizations studying online discussions (e.g., [12, 25]). Social network analysis (SNA) 2.2 Collaborative Design & Teacher Learning served as a primary research method for studying group in- Collaborative design is viewed as a form of professional de- teractions, characteristics of relations, and influence of these velopment [2, 54] and has been advocated as a desirable way relations in online teaching and learning [40, 46, 36]. For a for sustaining teachers to implement innovative practices en- CSCL process, the participants’ presence, roles, and their in- hanced by advanced technologies [2, 16, 34]. It is an activity teractions with other participants in the network are critical in which teachers and technology designers work together to factors that influence the collaboration process [35] and lead create teaching materials that comply with the function of to di↵erent levels of learning performance [11] or knowledge technologies, and the realities of teaching contexts [54]. It construction [1]. argues that active engagement, as well as the shared pro- cess of collaborative design, o↵ers ample opportunities for 3. METHODOLOGY teachers to reflect on and deepen their understanding of the 3.1 Research Questions usage of the new technology in classroom teaching [54]. The following three questions guide our analysis of the learn- ing process: RQ1 What are the interaction patterns of group The model of technological, pedagogical, and content knowl- participants (teacher-to-teacher)and knowledge construction edge (TPACK) [24] is frequently applied in this research (teacher to cognitive activities) during teachers’ collabora- field to describe what knowledge that teachers should de- tive design? RQ2 What are the sequential patterns of cogni- velop for technology integration. It consists of seven di- tive activities during the individual learning journey? RQ3 mensions: technology knowledge (TK), pedagogy knowl- What are the sequences of knowledge construction concern- edge (PK), content knowledge (CK), technology pedagogy ing discussion contents at the group level during teachers’ knowledge (TPK), technology content knowledge (TCK), collaborative design? pedagogical content knowledge (PCK), and technological pedagogical content knowledge (TPCK). Previous research 3.2 Research Context has identified two kinds of support necessary for develop- The data was collected from a two-week teacher learning ing teachers’ TPACK in collaborative design activities. One program conducted in April 2020. The program aims to is expert support, which means the design teams should in- equip teachers with sufficient knowledge about teaching with volve participants who are knowledgeable in the area of con- an ML-enhanced scientific discovery learning environment, tent, pedagogy, and technology on the materials that are be- SmileyDiscovery, designing for supporting STEM teaching ing developed [18]. The other is process support, referring to and learning in K-12 contexts [59]. In this study, we mainly monitoring the design process for ensuring the design inten- focused on the second session - teacher-as-designer in which tion is achieved [4, 27]. These studies employ a descriptive teachers worked collaboratively to design ML-enhanced les- analysis method, while our research examines the learning son plans by using SmileyDiscovery components (Fig. 6). process using statistics and visualization techniques. Eighteen teachers were divided into four groups (noted as group A, group B, group C, and group D) based on their 2.3 Learning Process Analysis in CSCL teaching grades and subjects. Each group included a par- Understanding the temporal aspect of learning is essential ticipant (for example, a1) volunteering for a mediator and as learning, by nature, is a process that occurs over time [23, a researcher(for example r1) playing as a facilitator (Ta- 41]. In the context of CSCL, two reasons stand out to study ble 1). Due to the COVID-19 lockdown, teachers commu- the temporal data of the collaboration process: 1) CSCL is a nicated with each other via ZOOM and created the lesson complex social process, including characteristics of multiple plans on design canvas supported by an online collaborative actors (e.g., learners, technology, etc.) between events over platform Lucidchart. The design canvas contains draggable time [26, 8]; 2) collaboration has a great potential to provide cards representing di↵erent SmileyDiscovery system compo- a learning environment with the shared learning process and nents (Fig. 6) for teachers to select for specific instructional steps in their lesson plan. The collaborative design activ- each turn of speech, reaching the agreement (Cohen’s Kappa ity consists of four phases: Deciding topic (10min)- teachers = 0.97). Second, we coded the cognitive activity for each select a subject matter to work on; Discussing learning ob- turn using an adapted version of the meta-cognitive regula- jectives (10min)- teachers identify the targeted grade levels tion coding scheme [21] (See Table 7, 8, 9). The original of students, questions of their interests, and other materials coding framework (see [21]) is developed by [53] to analyze required to fulfill the learning materials; Developing learning group knowledge construction behavior and validated in [48]. activities (25min)- teachers determine the pedagogical steps We extended it with response tokens (e.g. right, yeah, Uh according to the 5E instructional model [5], then design the huh, and hmm), showing that a talk sent by a speaker has related instructional activity in this step, and select the ap- been received by the audience). These response tokens are propriate SmileyDiscovery features that could support im- important for analyzing discussions since they serve to for- plementations of each instructional activity; Reflecting the ward the course of a conversation [33]. Four low-level codes, design (20min)- teachers critically reflect on the current les- thus, were generated after we conducted an open coding for son plan design and propose the desired improvement on a the transcripts: follow-up response (FU), show uncertainty specific aspect of SmileyDiscovery. (SU), show hearing (SH), agree with peers (AP). Two re- searchers independently coded all the transcripts, reaching The end product of teachers’ collaboration design are the de- a near-perfect agreement (Cohen’s kappa = 0.95). signed lesson plans that include specific instructional steps (e.g., Fig. 7) listed along with corresponding SmileyDiscov- For the one-mode analysis, we structured two data files record- ery components (Fig. 6). We assessed the quality of lesson ing 1) an edge list (all source-target directions and each tie plans as the group learning outcome using an empirically weight) and 2) a node list (all participant id and their roles) validated framework [17] (see Table 5). It was created for of the network. The weight of each turn is assigned accord- measuring the quality of technology-enhanced teaching ma- ing to the level of cognitive activity: high-level (value = 2) terials built from the TPACK model [24]. Two researchers and low-level (value = 1). A two-dimensional co-concurrence independently evaluated the lesson plans, achieving a near- matrix was constructed for the two-mode analysis, calcu- perfect agreement (Cohen’s kappa = 0.92). lating each participant’s participation frequency engaged in each type of cognitive activity. The measures of the social network analysis are shown in the Appendix (Table 3). Table 1: Demographic information for each group. Group Grades Subjects Researcher Mediator Scores A B C Elementary (N=3), Middle school (N=1) Middle school (N=4) High school (N=4), Middle school (N=1) Science (N=2), Math (N=2) Science (N=3), Math (N=1) Science (N=1), Math (N=4) r1 r2 r3 a1 b4 c4 3.50 3.17 3.80 3.3.2 RQ2: Sankey Diagram D High school (N=5) Science (N=5) r4 d3 3.78 A Sankey diagram is a visualization tool that illustrates quantitative information of the activity flow of individual 3.3 Data and Analytical Approach participants by using directed, and weighted graphs [42]. We collected recordings of four groups’ collaborative design Thus, we applied it to discern the patterns of cognitive activ- and transcribed the verbal data for analysis. The raw tran- ity flow each participant engaged in across di↵erent phases scripts contain 1869 turns. The social talk and incomplete of the collaborative design. Moreover, we can explore the se- talk were dropped o↵ as they are less relevant with knowl- quential patterns of teachers’ engagement and role-switching edge construction, ended with 1765 turns in total (Group A in di↵erent types of cognitive activities. To simplify the vi- = 504, Group B = 328, Group C = 478, Group D = 455). sualization, we grouped all cognitive activities into six cate- gories based on the purposes of learning: plan the next step 3.3.1 RQ1: Social Network Analysis (pl, ph), evaluate the design purpose (el, eh), enhance the A social network analyzes the patterns of connections (repre- group’s conceptual understanding (vm, ei, jd, rm, sm, qm), sented as ties or edges with strengths and directions) among seek or provide basic information (si, ai), follow up with- entities (individual, groups, events, etc.), represented by out creating much new information (sh, su, ci, fu, ap), and nodes with sizes, and relations between entities [40, 46]. conclude an episode of discussion (cd, sd). And the x-axis This research question investigates participants’ positions represents the sequence of a participant’s cognitive behav- and their interactions in groups and their engagement in iors (e.g., one node with x = 7 represents the 7th cognitive di↵erent cognitive activities involved in the knowledge con- activity a participant conducted). struction process. Thus, we utilized SNA to visualize the relations and participants’ roles in the network and quanti- 3.3.3 RQ3: Lag Sequential Analysis fied the relations using both the node-level measures and Lag sequential analysis (LSA) is an analytical approach used network measures with the Igraph library in the R pro- for determining if a statistically significant dependence ex- gramming language. Conceptually, a social network can be ists between sequential events [3]. Many researchers have structured as a one-mode network [30] and a two-mode net- adopted it to understanding the sequential patterns of par- work with mode referred to the set of nodes [46]. One-mode ticipants’ behaviors in learning activities and what the de- analysis is used to study the relations of people (e.g., in- sired patterns would be for learning [55, 56]. We applied it teractions between teachers and participant’s positions and to explore the sequential patterns of knowledge construction roles). Two-mode analysis is used to analyze networks that occurring in the discussion contents that di↵erent groups en- involve participants and events (e.g., teacher’s participation gaged in collaborative design activities. frequency engaged in the knowledge construction process). We first chunked the transcripts into segments, whereby Before running the analysis, we segmented the transcripts to each segment corresponded to a unique topic of conversation the turn level. First, two researchers reviewed each group’s related to the design contents. For example, teachers were transcript independently to code the source and target of required to identify the learning objectives of the design les- son plan. A conversation around it, from the initiation to the end, is considered a topic. Second, we adapted the TPACK model [24] to code the knowledge dimensions shown by the specific speech of a turn (Table 6). Two researchers inde- pendently coded the TPACK and reached an almost perfect agreement, Cohen’s kappa = 0.95. Third, since we are in- terested in understanding the sequential pattern of di↵erent knowledge dimensions for each topic of conversation, the duplicated codes were dropped o↵ for each segment. For ex- ample, if TK occurs several times in one segment, we only counted it occurred once. The LSA is performed for each group using the program Generalized Sequential Query (GSEQ) [3]. First, we run the Pearson chi-square test to check if a significant dependence exists between knowledge dimensions. Then, we used the program to calculate the adjusted residual between any two knowledge dimensions. Figure 1: One-mode sociogram of teacher-teacher interac- tions.Top left: group A; Top right: group B; Bottom Left: 4. RESULTS group C; Bottom right: group D 4.1 RQ1.1 Teacher-teacher interaction Table 2: Network attribute of four groups. The sociogram of the teacher-teacher interaction (see Fig. 1) group A group B group C group D showed di↵erent roles of participants (researcher, mediator, No. of participants 5 5 6 6 No. of ties 797 668 1079 1181 teacher) and their levels of contributions to the discussion, Average degree of group 320 269 360 394 demonstrated by the position and size of nodes in the net- SD of Degree centrality 12.40 13.50 26.44 21.05 work. First, for the researcher position, r1(group A) and Density 39.85 33.40 36.00 39.37 Reciprocity 0.75 0.66 0.66 0.57 r4 (group D) had a higher degree of centrality (especially No. of nodes in cognitive activity 18 18 18 19 out-degree centrality) than r2 (group B) and r3 (group C), No. of low-level cognitive activity 314 206 263 256 with r2 holding the least out-degree centrality, evident by % of low-level cognitive activity 62.30% 62.80% 55.02% 56.26% No. of high-level cognitive activity 190 122 215 199 the node-level measures (see Appendix Table. 4). This il- % of high-level cognitive activity 37.70% 37.20% 44.98% 43.74% lustrated that r1 and r4 played more proactive roles in fa- cilitating the discussion and o↵ering guidance, whereas r2 intervened less and relied more on the mediator b4 to fa- Compared the one-node network attributes with the scores cilitate the discussion. Second, for the mediator position, of teacher-designed lesson plans, group C and group D with d3 (out-degree centrality = 500) and c4 (out-degree central- high avg degrees and high density and more proactive roles ity = 469) played dominant roles in the group discussions, of researcher and mediator had better final sores. Although taking responsibility for note-taking and guiding the discus- not tested statistically, the association might demonstrate sion than a1 (out-degree centrality = 215) and b3(out-degree a need for explicit facilitation and mediation training of re- centrality = 260). Third, compared the out-degree central- searchers and mediators in the future. ity for teacher participants and visual positions in the net- work, b1, c5, and d4 are relatively peripheral in contribut- ing to the group discussions. The reason might be due to 4.2 RQ1.2 Teacher-cognitive activity interac- the teacher’s insufficient technology knowledge about Smi- tion leyDiscovery. As to the closeness measure, we observed the The two-mode network attributes Table 2 showed the dis- highest closeness of participants in group A, demonstrating tribution of low and high-level cognitive activity frequency their relatively equal participation and greater mutuality in during teachers’ collaborative design. While group A had the discussions. the highest number of cognitive activity events (N = 504), the ratio of engagement in the high-level cognitive activities Comparing the one-mode network attribute (see Fig. 2), we (37.70% ) was smaller than that of Group C (44.98%) and found all groups shared a relatively high density value, be- group D (43.74%). Group B had the slightest participation tween 30-40%. This indicated highly active and cohesive in cognitive activities (N = 328) and the high-level cognitive participation in the group discussion across four groups, with activities (ratio = 37.20%). This indicates that the better no isolated participants. On average, group C (avg.degree quality of the lesson plans designed by group C and group D = 360) and group D (avg.degree = 394) had a higher fre- might result from the high-quality discussion they engaged quency of group interactions than group A (avg.degree = in the knowledge construction process. 320) and group B (avg.degree = 268). However, the contri- bution among participants was rather equal for group A and The two-mode sociogram (Fig. 2) shows interaction patterns group B, indicated by the smaller standard deviation(SD) of the participants with the cognitive activities, with the of the degree centrality-12.40 and 13.50 respectively- com- size of the participant node indicating the degree of par- pared to that of group C (SD = 26.44) and group D (SD ticipation. Group A had a relatively equal distribution of = 21.05). Reciprocity refers to the balance of the network. cognitive activities. Participants b1 and c5 had the rela- The values of all groups are larger than 0.5, showing a rel- tively less engagement compared to the other participants atively high mutual communication between participants. in their groups. Group D was unequal as the mediator Figure 3: The Sankey diagram illustrating 18 teachers’ prepa- ration phase during the design activity. Figure 2: Two-mode sociograms of participant to cognitive activity interactions.Blue circle: Participants (m - mediator); Red square: Cognitive activity. Top left: group A; Top right: group B; Bottom Left: group C; Bottom right: group D. d3 undertook more cognitive activities than other partici- pants.Furthermore, participants of group C shared a rela- tively higher frequency of the high-level cognitive activity; because the nodes of cognitive activities are near each other and surround all participant nodes. The researcher and me- diator in group C both actively participated in high-level planning (PH) and explore ideas (EI). For group D, we ob- served a proactive role of mediator in facilitating discus- sions, demonstrated by d4’s engagement in the high-level cognitive activities such as QM- Questioning, RM- reflect meaning, and PH- high-level planning. Also, the mediator Figure 4: The Sankey diagrams illustrating the phases for engaged in low-level cognitive activities for responding to learning objective setup, learning activity development and the participants and transitioning the discussion, as SI- seek reflection. information, SD- stop discussion, and SU- show uncertainty. We argue that for future study, the researcher or mediator should be o↵ered a more explicit strategy to help facilitate generated much longer cognitive activity flows than other the discussion and actively participate in the collaborative participants (long-tail flows). Those participants who con- design to support participants engaging in high-level cogni- tributed more to the discussions tended to be more con- tive activities. stantly and frequently engaged in cognitive activities that enhance the group’s conceptual understanding (‘enhance’). This indicates that individual participant’s frequency of the 4.3 RQ2 Sequential patterns of cognitive ac- cognitive activities might be positively related to one’s con- tivities tribution to the group’s conceptual understanding. Preparation phase. Di↵erent from the rest phases, there is no enhance of conceptual understanding during this phase 4.4 RQ3 Sequential patterns of TPACK tran- (Fig. 3). There are two main patterns identified in partici- sition in knowledge construction pants’ cognitive flows: (1) some participants mainly focused Fig. 5 illustrates the patterns of TPACK transition in knowl- on seeking information or providing information; (2) some edge construction of di↵erent groups. The curve indicates participants (e.g., researchers) illustrated how the next step the sequence between two knowledge dimensions and the should take place and evaluated the current progress. values are the adjusted residual of the sequential transition. Any sequential transition below 1.96 is dropped o↵ as it indi- cates a non-significance between two knowledge dimension. The phases for learning objective setup, learning activ- Also, the information of group B is not presented here be- ity development and reflection. During these three phases cause the overall pattern is not significant assessed by the (Fig. 4), most participants switched frequently between the chi-sqaure test (p = 0.97). By comparing the knowledge role to enhance the group’s conceptual understanding (‘en- construction patterns of group A, group C, and group D hance’), the role to seek or add basic information (‘info’), with the quality of their design artifacts, we found the level and the role to follow up with someone else (‘follow’). Fig. 4 of complexity of TPACK transition patterns in knowledge showed that, during these three phases, a few participants construction corresponds to the groups’ outcomes. ing the discussions and taking notes for the whole group, demonstrated by their engagement in planning, evaluation, and direction while group A had a relatively equal engage- ment in the cognitive activities. Previous literature showed that assigning students with leadership roles (e.g., facilita- tors) could empower students to engage in the discussions [39], which echoed with the case in group C and group D, but how to empower and engage the peripheral members would be a future discussion. Also, discussion facilitation strategies make a significant di↵erence in the extent of col- laboration [40, 49]. In our case, researchers who take an active role in facilitation and help address the technological and content knowledge gap of participants promoted bet- ter quality of discussion. Thus we need to explicitly train mediators and researchers to facilitate discussions. Figure 5: Patterns of knowledge construction in four groups produced by lag sequential analysis. Engage learners in more meaningful discussion. The Sankey diagrams show that teachers who participated in the discussion constantly and frequently engaged in more activ- Group C (with the highest group outcome) shows the most ities enhancing the group’s conceptual understanding. One complex patterns of knowledge transition, including six sets interpretation is that note-takers in each group who had to of transitions with six knowledge dimensions involved (TK– talk more throughout the design activity needed to take re- >TPK, CK–>TCK, CK–>PCK, TPK–>TK, PCK–>CK, sponsibility for the learning activity construction and reflec- PCK–>PK). Group D (the second-highest group outcome) tion; in turn, they got involved in more cognitive activities consists of four sets of transitions with five knowledge dimen- that enhance the conceptual understanding. Another po- sions involved (PCK–>PK, PK–>TPK, TCK–>CK, CK– tential explanation is that participants who produced more >TCK). Compared with group C, group D lacks the tran- dialogues of “enhance” had more opportunities to explore sition between TK with any other knowledge dimensions. their ideas further. This suggests the facilitation is needed That means teachers in group D were more likely to discuss to prompt learners with fewer discourses or fewer “enhance” TK exclusively for a design component than connecting it cognitive activities to share their ideas with the group. to others. Group A (the third highest group outcomes) has a simpler pattern, containing four sets of knowledge con- struction with four knowledge dimensions (TK–>TPK, CK– >TCK, TPK–>TK, TCK–>CK). Compared with group C Knowledge transition in group discussions contributes and group D, group A lacks PK and PCK as well as their to the TPACK development. The results of the lag sequen- tial analysis suggest that the transitions between the sub- transitions to other knowledge dimensions. Group B (the dimensions of TPACK in collaborative design might con- lowest group outcomes) does not show significant depen- tribute to groups’ learning outcomes. This finding adds new dence between any two knowledge dimensions. According to evidence to the research of TPACK, showing that grasping the LSA results, the higher group outcomes, the more com- the connections between TK, PK, and CK is significant for plex TPACK transition patterns displayed in the knowledge developing an integrative understanding of TPACK. Previ- construction process. Nevertheless, given the small sample ous research has found the impacts of TK, PK, and CK on data, further studies are needed to validate this finding. teachers’ TPACK by a regression model using pre-post as- sessments [6]. Few studies, however, have examined it taking 5. DISCUSSION a process perspective. Given the importance of collaborative design in developing TPACK [24], our research suggests the Effective group discussion and the role of participants. design contents are better to be addressed by discussing the Group C and group D outperformed group A and group B knowledge dimensions and the related sub-dimensions. For as for the discussion quality, and the result is potentially as- example, when teachers are engaged in talking about TPK sociated with the previously-graded design artefacts scores, for a while, the facilitator can intervene and guide teach- with lesson plan scores (group C > group D > group A ers to discuss TK or/and PK related to the TPK. Such a > group B). The result, consistent with the previous liter- process provides teachers with opportunities to understand ature, indicates that groups engaged in a large amount of each dimension of TPACK and its connections. Our further high-level conceptual understanding, elaboration, and justi- step is to conduct a qualitative analysis of the transcript fication of content material were also associated with better data, primarily how knowledge transition occurred in some overall conceptual understanding demonstrated in the end interaction units but not others. The findings generated can learning product [21]. We expected to see high group den- o↵er more insights on how to facilitate teachers’ collabora- sity, active participation, and high-level knowledge construc- tive design activities. tion To promote a high-quality discussion [22, 53]. 6. REFERENCES The mediator and the researcher might play a pivotal role [1] R. Aviv, Z. Erlich, G. Ravid, and A. Geva. Network in promoting the discussion quality. Group C and group analysis of knowledge construction in asynchronous D’s mediators (c4 and d3) played a dominant role in direct- learning networks. Journal of Asynchronous Learning Networks, 7(3):1–23, 2003. [16] A. Handelzalts. Collaborative curriculum development [2] M. A. Bakah, J. M. Voogt, and J. M. Pieters. in teacher design teams. In Collaborative curriculum Updating polytechnic teachers’ knowledge and skills design for sustainable innovation and teacher learning, through teacher design teams in ghana. Professional pages 159–173. Springer, Cham, 2019. development in education, 38(1):7–24, 2012. [17] J. Harris, N. Grandgenett, and M. Hofer. Testing a [3] R. Bakeman and V. Quera. Sequential analysis and tpack-based technology integration assessment rubric. observational methods for the behavioral sciences. In Society for Information Technology and Teacher Cambridge University Press, 2011. Education International Conference, pages 3833–3840. [4] H. Becuwe, J. Tondeur, N. Pareja Roblin, J. Thys, Association for the Advancement of Computing in and E. Castelein. Teacher design teams as a strategy Education (AACE), 2010. for professional development: The role of the [18] T. Huizinga, A. Handelzalts, N. Nieveen, and J. M. facilitator. Educational Research and Evaluation, Voogt. Teacher involvement in curriculum design: 22(3-4):141–154, 2016. Need for support to enhance teachers’ design expertise. [5] R. W. Bybee, J. A. Taylor, A. Gardner, Journal of curriculum studies, 46(1):33–57, 2014. P. Van Scotter, J. C. Powell, A. Westbrook, and [19] G.-J. Hwang, H. Xie, B. W. Wah, and D. Gašević. N. Landes. The BSCS 5E instructional model: Origins Vision, challenges, roles and research issues of artificial and e↵ectiveness. BSCS, 5, 88-98, Colorado Springs, intelligence in education, 2020. Co, 2006. [20] A. Khanlari, M. Resendes, G. Zhu, and [6] C. S. Chai, J. H. L. Koh, and C.-C. Tsai. Facilitating M. Scardamalia. Productive knowledge building preservice teachers’ development of technological, discourse through student-generated questions. pedagogical, and content knowledge (tpack). Journal Philadelphia, PA: International Society of the of Educational Technology & Society, 13(4):63–73, Learning Sciences., 2017. 2010. [21] D. K. Khosa and S. E. Volet. Productive group [7] B. Chen, M. Resendes, C. S. Chai, and H.-Y. Hong. engagement in cognitive activity and metacognitive Two tales of time: uncovering the significance of regulation during collaborative learning: Can it sequential patterns among contribution types in explain di↵erences in students’ conceptual knowledge-building discourse. Interactive Learning understanding? Metacognition and Learning, Environments, 25(2):162–175, 2017. 9(3):287–307, 2014. [8] B. Chen, A. F. Wise, S. Knight, and B. H. Cheng. [22] A. King. Structuring peer interaction to promote Putting temporal analytics into practice: the 5th high-level cognitive processing. Theory into practice, international workshop on temporality in learning 41(1):33–39, 2002. data. In Proceedings of the sixth international [23] S. Knight, A. F. Wise, and B. Chen. Time for change: conference on learning analytics & knowledge, pages Why learning analytics needs temporal analysis. 488–489, 2016. Journal of Learning Analytics, 4(3):7–17, 2017. [9] T. H. Chiang, S. J. Yang, and G.-J. Hwang. Students’ [24] M. Koehler and P. Mishra. What is technological online interactive patterns in augmented reality-based pedagogical content knowledge (tpack)? inquiry activities. Computers & Education, 78:97–108, Contemporary issues in technology and teacher 2014. education, 9(1):60–70, 2009. [10] B. Collis and J. Moonen. Contribution-oriented [25] J. Lämsä, R. Hämäläinen, M. Aro, R. Koskimaa, and pedagogy. In Encyclopedia of Distance Learning, S.-M. Äyrämö. Games for enhancing basic reading and Second Edition, pages 439–446. IGI Global, 2009. maths skills: A systematic review of educational game [11] M. S. Despotović-Zrakić, A. B. Labus, and A. R. design in supporting learning by people with learning Milić. Fostering enginering e-learning courses with disabilities. British Journal of Educational Technology, social network services. In 2011 49(4):596–607, 2018. 19thTelecommunications Forum (TELFOR) [26] J. Lämsä, R. Hämäläinen, P. Koskinen, J. Viiri, and Proceedings of Papers, pages 122–125. IEEE, 2011. E. Lampi. What do we do when we analyse the [12] N. Ding, J. Wei, and M. Wolfensberger. Using temporal aspects of computer-supported collaborative epistemic synchronization index (esi) to measure learning? a systematic literature review. Educational students’ knowledge elaboration process in cscl. Research Review, page 100387, 2021. Computers & education, 80:122–131, 2015. [27] T. H. Levine and A. S. Marcus. How the structure and [13] I. Evangelista, G. Blesio, and E. Benatti. Why are we focus of teachers’ collaborative activities facilitate and not teaching machine learning at high school? a constrain teacher learning. Teaching and teacher proposal. In 2018 World Engineering Education education, 26(3):389–398, 2010. Forum-Global Engineering Deans Council [28] J.-W. Lin, L.-J. Mai, and Y.-C. Lai. Peer interaction (WEEF-GEDC), pages 1–6. IEEE, 2018. and social network analysis of online communities [14] L. C. Freeman. Centrality in social networks with the support of awareness of di↵erent contexts. conceptual clarification. Social networks, 1(3):215–239, International Journal of Computer-Supported 1978. Collaborative Learning, 10(2):139–159, 2015. [15] R. Hämäläinen and K. Vähäsantanen. Theoretical and [29] Z. Liu, L. Kang, M. Domanska, S. Liu, J. Sun, and pedagogical perspectives on orchestrating creativity C. Fang. Social network characteristics of learners in a and collaborative learning. Educational Research course forum and their relationship to learning Review, 6(3):169–184, 2011. outcomes. In CSEDU (1), pages 15–21, 2018. [30] D. A. Luke. A user’s guide to network analysis in R. participation and social dimensions of Springer, 2015. computer-supported collaborative learning through [31] J. Malmberg, S. Järvelä, H. Järvenoja, and social network analysis: which method and measures E. Panadero. Promoting socially shared regulation of matter? International Journal of Computer-Supported learning in cscl: Progress of socially shared regulation Collaborative Learning, 15(2):227–248, 2020. among high-and low-performing groups. Computers in [46] J. Scott and P. J. Carrington. The SAGE handbook of Human Behavior, 52:562–572, 2015. social network analysis. SAGE publications, 2011. [32] L. S. Marques, C. Gresse von Wangenheim, and J. C. [47] F. Sullivan, E. Suárez, E. Pektas, and L. Duan. HAUCK. Teaching machine learning in school: A Developing pedagogical practices that support systematic mapping of the state of the art. disciplinary practices when integrating computer Informatics in Education, 19(2):283–321, 2020. science into elementary school curriculum. 2020. [33] M. McCarthy. Talking back:” small” interactional [48] M. Summers and S. Volet. Group work does not response tokens in everyday conversation. Research on necessarily equal collaborative learning: evidence from language and social interaction, 36(1):33–63, 2003. observations and self-reports. European Journal of [34] S. McKenney, F. Boschman, J. Pieters, and J. Voogt. Psychology of Education, 25(4):473–492, 2010. Collaborative design of technology-enhanced learning: [49] J. Thormann and P. Fidalgo. Guidelines for online what can we learn from teacher talk? TechTrends, course moderation and community building from a 60(4):385–391, 2016. student’s perspective. Journal of Online Learning & [35] C. S. Ng, W. S. Cheung, and K. F. Hew. Interaction Teaching, 10(3):374–388, 2014. in asynchronous discussion forums: peer facilitation [50] R. Tirado, Á. Hernando, and J. I. Aguaded. The e↵ect techniques. Journal of Computer Assisted Learning, of centralization and cohesion on the social 28(3):280–294, 2012. construction of knowledge in discussion forums. [36] M. Oliveira and J. Gama. An overview of social Interactive Learning Environments, 23(3):293–316, network analysis. Wiley Interdisciplinary Reviews: 2015. Data Mining and Knowledge Discovery, 2(2):99–115, [51] A. Traxler, A. Gavrin, and R. Lindell. Networks 2012. identify productive forum discussions. Physical Review [37] T. Opsahl, F. Agneessens, and J. Skvoretz. Node Physics Education Research, 14(2):020107, 2018. centrality in weighted networks: Generalizing degree [52] A. Vazhayil, R. Shetty, R. R. Bhavani, and N. Akshay. and shortest paths. Social networks, 32(3):245–251, Focusing on teacher education to introduce ai in 2010. schools: Perspectives and illustrative findings. In 2019 [38] F. Ouyang. Using three social network analysis IEEE Tenth International Conference on Technology approaches to understand computer-supported for Education (T4E), pages 71–77. IEEE, 2019. collaborative learning. Journal of Educational [53] S. Volet, M. Summers, and J. Thurman. High-level Computing Research, page 0735633121996477, 2021. co-regulation in collaborative learning: How does it [39] F. Ouyang and Y.-H. Chang. The relationships emerge and how is it sustained? Learning and between social participatory roles and cognitive Instruction, 19(2):128–143, 2009. engagement levels in online discussions. British [54] J. Voogt, T. Laferriere, A. Breuleux, R. C. Itow, D. T. Journal of Educational Technology, 50(3):1396–1414, Hickey, and S. McKenney. Collaborative design as a 2019. form of professional development. Instructional [40] F. Ouyang and C. Scharber. The influences of an science, 43(2):259–282, 2015. experienced instructor’s discussion design and [55] S.-Y. Wu and H.-T. Hou. How cognitive styles a↵ect facilitation on an online learning community the learning behaviors of online problem-solving based development: A social network analysis study. The discussion activity: A lag sequential analysis. Journal Internet and Higher Education, 35:34–47, 2017. of educational computing research, 52(2):277–298, [41] P. Reimann. Time is precious: Variable-and 2015. event-centred approaches to process analysis in cscl [56] T.-C. Yang, S. Y. Chen, and G.-J. Hwang. The research. International Journal of Computer-Supported influences of a two-tier test strategy on student Collaborative Learning, 4(3):239–257, 2009. learning: A lag sequential analysis approach. [42] P. Riehmann, M. Hanfler, and B. Froehlich. Computers & education, 82:366–377, 2015. Interactive sankey diagrams. In IEEE Symposium on [57] S. Zhang, Q. Liu, W. Chen, Q. Wang, and Z. Huang. Information Visualization, 2005. INFOVIS 2005., Interactive networks and social knowledge pages 233–240. IEEE, 2005. construction behavioral patterns in primary school [43] T. K. Rogat and L. Linnenbrink-Garcia. Socially teachers’ online collaborative learning activities. shared regulation in collaborative groups: An analysis Computers & Education, 104:1–17, 2017. of the interplay between quality of social regulation [58] Y. Zhang, J. Wang, F. Bolduc, W. G. Murray, and and group processes. Cognition and Instruction, W. Sta↵en. A preliminary report of integrating science 29(4):375–415, 2011. and computing teaching using logic programming. In [44] M. Saqr and A. Alamro. The role of social network Proceedings of the AAAI Conference on Artificial analysis as a learning analytics tool in online problem Intelligence, volume 33, pages 9737–9744, 2019. based learning. BMC medical education, 19(1):1–11, [59] X. Zhou, J. Tang, M. Daley, S. Ahmad, and Z. Bai. 2019. “now, i want to teach it for real!”: Introducing [45] M. Saqr, O. Viberg, and H. Vartiainen. Capturing the machine learning as a scientific discovery tool for k-12 Figure 6: SmileyDiscovery system components for teachers to drag and drop during the collaborative design of the ML- empowered scientific discovery lesson plan: (a) components to introduce the STEM context and multidimensional feature space, modify and view the data-face visualization mapping; (b) components to facilitate pattern interpretation (i.e., intra- cluster pattern, inter-cluster pattern); (c) components to fa- cilitate the conduction of clustering (i.e., manual clustering, automatic clustering). teachers. 2021. (In press). APPENDIX Figure 7: The instructional steps designed in the ML- empowered scientific discovery lesson plan created by one group of teachers collaboratively (group A). Table 3: Node-level and network-level measures for Social Network Analysis Measures Definition Level In-degree centrality Total number of interactions a participant received from others in the network [14]. High value indicates a partici- Node-level pant’s prestige or influence when engaging in a discussion [29, 44] Out-degree centrality Total number of interactions a participant sent to others in the network. High value indicates a participant active- ness in providing comments or information to others in the network [38, 45]. Closeness centrality Length of paths from a participant to all others in the network, defined as the inverse total length [37]. High value indicates high efficiency of a participant has on re- ceiving and spreading information in the network. Number of participants Number of participants in the network Number of ties Total number of ties in a network without tie weights. Average degree by group Average number of the sum of connections of a group. One-mode Network Density Ratio of the number of ties observed in the network di- vided by the maximum number of possible ties (equals to n*(n-1), where n is the number of nodes. For example, our max number of possible ties will be 30). High value indicated the high level of teacher participation in the discussion. Reciprocity Likelihood of nodes in a directed network to be mutually linked [46]. It indicates the balanced of the mutual dyads relations in a network, and reflects teacher participants’ connection level within the network. Number of nodes in cogni- Number of cognitive activity types in the network. tive activity Two-mode Network Number and ratio of low- Number and ratio of low-level cognitive activities in the level cognitive activity fre- network. quency Number and ratio of high- Number and ratio of high-level cognitive activities in the level cognitive activity fre- network. quency Table 4: Node-level attribute of four groups. Participant Roles In-degree Out-degree Closeness ID Centrality Centrality a1 mediator 169 215 0.80 a2 teacher 166 115 0.67 a3 teacher 168 110 0.57 a4 teacher 154 89 0.57 r1 researcher 140 267 0.50 b1 teacher 122 22 0.25 b2 teacher 156 140 0.25 b3 teacher 133 75 0.17 b4 mediator 133 260 0.25 r2 researcher 124 171 0.25 c1 teacher 177 175 0.13 c2 teacher 161 98 0.20 c3 teacher 198 120 0.17 c4 mediator 224 469 0.10 c5 teacher 156 34 0.14 r3 researcher 163 183 0.20 d1 teacher 191 86 0.20 d2 teacher 228 172 0.20 d3 mediator 209 500 0.20 d4 teacher 172 22 0.17 d5 teacher 204 93 0.20 r4 researcher 177 308 0.17 Table 5: Assessment rubric for teacher-designed ML-enhanced lesson plans adapted from Harris et al (2010). Criteria 4 3 2 1 Learning Technologies selected Technologies selected Technologies selected Technologies selected contents and for use in the les- for use in the lesson for use in the les- for use in the lesson Technologies son plan are strongly plan are aligned with son plan are partially plan are not aligned (TCK) aligned with one or one or more learning aligned with one or with one or more more learning goals. goals. more learning goals. learning goals. Instructional Technologies selected Technologies selected Technologies selected Technologies selected Strategies and optimally supports in- supports instructional minimally supports in- not supports instruc- Technologies structional strategies. strategies. structional strategies. tional strategies. (TPK) Instructional Instructional strate- Instructional strate- Instructional strate- Instructional strate- Strategies gies selected are gies selected are gies selected are gies selected are and Learning exemplary for the appropriate, but not marginally appropriate inappropriate for the Contents learning goals. exemplary for the for the learning goals. learning goals. (PCK) learning goals. Overall calculated as an average of the TCK, TPK, and PCK. (TPCK) Table 6: Codes of knowledge dimensions of TPACK in group discussions. Codes Knowledge Dimension Description TK Technology knowledge Teachers talked about the concepts and the usage of SmileyDiscovery and ML techniques. PK Pedagogical knowledge Teachers talked about issues related to instructional methods, class- room management, and students’ characteristics. CK Content knowledge Teachers talked about the details of learning activities related to the content topic. TPK Technological Pedagogical Teachers talked about how to sca↵old scientific discovery through ap- Knowledge plying SmileyDiscovery and ML techniques, and what aspects of Smi- leyDiscovery can be improved to provide better pedagogical support. TCK Technology Content Knowl- Teachers talked about how to facilitate student content learning by edge using SmileyDiscovery and ML techniques, and what aspects of Smi- leyDiscovery should be improved to fulfill the content learning. PCK Pedagogical Content Knowl- Teachers talked about how to facilitate student content learning by con- edge sidering such pedagogical aspects as instructional methods, classroom management, and students’ characteristics. TPCK Technological pedagogical Teachers talked about the alignment of SmileyDiscovery, instructions content knowledge of scientific discovery activity, and subject matters. Table 7: Coding scheme for cognitive activities during the group regulation of planning. Codes Abbr. Level Description Planning without justification PL Low Determine how to achieve the task; Assign tasks to certain participants; Set- Planning with justification PH High ting of ground rules and norms for group discussion; Guide the activity flow. Table 8: Coding scheme for cognitive activities during the group regulation of monitoring. Codes Abbr. Level Description Seek information SI Ask for more factual information to assist with the group’s current under- standing of the task or content, often a tentative enquiry. Add information AI Inject new factual information to bring the group back into gathering facts or pursuing content discussion. This may also include adding information that Low was previous discussed. Agree with peers AP Agree with someone’s proposal and didn’t know the answer before previous person’s input. Stop Discussion SD Stop the flow of discussion to bring the group to a decision or action point. It could trigger the end of an episode of high-level talk. Follow-up response FU A brief response from the talk initiator. Confirm information CI Provide a confirmation with knowing the answer before previous person’s in- put, following a question. Show uncertainty SU Express uncertainty about the content mentioned in the previous conversation. Show hearing SH Show a sign of hearing others’ response, but not with clear semantic meaning. Seek meaning SM Ask questions that would enhance the group’s conceptual understanding of the case. Volunteer meaning VM Propose an explanation, elaboration, or interpretation that enhance the group’s conceptual understanding of the case. It could be based on knowl- High edge and understandings from readings or experience. Explore Ideas EI Engage in tentative explanations, interpretations or speculations to enhance the group’s conceptual understanding of the case (on their own ideas). Question meaning QM Question the group’s current conceptual understanding of aspects of the case with a view to clarify or rectify that understanding. Justify decision JD Justify a task-related decision on the basis of the group’s conceptual under- standing of the case. Reflect on meaning RM Reflect on the group’s current understanding of the content or case and what is needed to further enhance understanding. Conclude from discussions CD Draw a summary or conclusion from the discussion. Table 9: Coding scheme for cognitive activities during the group regulation of evaluation. Codes Abbr. Level Description Evaluate without justification EL Low Check if the requirements have been met, if the contents on the design canvas Evaluate with justification EH High match with what participants talk about, if anyone has inquiries, if everyone can follow.