Opening the Black Box of Practice-Based Learning: Human- Centred Design of Learning Analytics Nina Valkanova 1, Mutlu Cukurova 2, Annelie Berner1, Katerina Avramides2, Manolis Mavrikis2, 1 Copenhagen Institute of Interaction Design n.valkanova, a.berner@ciid.dk; 2 UCL Knowledge Lab, University College London, m.cukurova, k.avramides, m.mavrikis@ucl.ac.uk Abstract: Practice-based learning activities are an important aspect of education, particularly for science, technology, engineering and mathematics (STEM) subjects. Their immense importance to STEM curricula is unequivocal and so are the teachers’ and students’ need for support during those activities. However, considering the open-ended and hands-on nature of practice-based learning activities, designing, deploying and validating learning analytics visualisations remains as a significant challenge for the state- of-the-art learning analytics. In this paper, we present our human- centered contextual enquiry approach for generating requirements and its preliminary results in the form of visualisations that have the potential to support facilitators and learners. Although there have been certain attempts to provide learning analytics for increasing awareness, supporting reflection and facilitating decision-making and intervention, to our knowledge, our research presented is the first attempt to provide such information regarding students’ progress during practice-based learning activities. Keywords: practice-based learning, contextual inquiry, learning analytics, feedback, visualizations Introduction and Background In STEM teaching, practice-based learning is considered to be an essential part of teaching and learning (Millar, 2004). Guidance is essential in those activities (Clark, 2009), since allowing students to work independently does not always lead to meaningful learning outcomes (Cukurova & Bennett, 2014). Facilitators of practice- based learning activities, as well as learners, are in need of tools that can provide them with indicators of learning processes in order to support teacher monitoring and learner self-reflection and self-regulation (Dillenbourg et al., 2011). Yet, little is known about what should be and can be presented to teachers and learners in practice-based learning environments. In this paper, we present our visualisation tool based on the outcomes of a contextual inquiry in practice-based STEM teaching and human-centred iterative design methodology. Learning Visualizations A number of visualisation tools have been developed for online, face-to-face, and blended learning settings, where this data is more readily available. Most of these attempts aim to support teachers (Verbert et al., 2014), but some applications have also been developed to support students’ awareness and self- reflection (e.g. (Govaerts, Verbert, Klerkx, & Duval, 2010). Researchers have developed visualisations of students’ access to resources, their communication patterns in forums, as well as frequenc y and timings of their activities (e.g. (Coffrin, Corrin, de Barba, & Kennedy, 2014). Such visualisations enable teachers to provide better support, for example by identifying patterns of participation and intervening in problematic groups (Van Leeuwen, Janssen, Erkens, & Brekelmans, 2014). Similarly, student learning in intelligent tutoring systems is more easily tracked, and several visualisation tools have been developed to provide students with information on their progress e.g. (Lafford, 2004). These solutions, however, do not necessarily transfer in open- ended, practice-based learning where the technical challenges are very different and the usability and pedagogical requirements are not yet well understood. First, practice-based learning activities usually take place simultaneously in multiple groups of students, sometimes in a range of physical spaces and across a large time- span. In addition, the diverse set of digital and non-digital activities cannot always be tracked keeping practice-based learning largely out of the scope of current learning analytics trends, despite its immense importance to STEM curricula. We are interested in investigating whether learning analytics can support the challenging role of the teacher or facilitator in such settings and/or help students reflect on their own practice. The challenges teachers face during practice-based learning, particularly in formal education, are well documented. 40 Copyright © 2016 for this paper by its authors. Copying permitted for private and academic purposes. Teachers are rarely aware of the processes followed by students during these activities (Race, 2001), and it is challenging for them to provide appropriate support to individual students, who have different needs, strengths and weaknesses (Zhang, Zhao, Zhou, & Nunamaker, 2004). Teachers can only be aware of what a small number of students are doing at any one time in the classroom. It is, therefore, hard for teachers to know which students are making progress, and which are in difficulty and in need of additional support. It is a challenge for teachers to understand the process by which students have arrived at the current state of their practice- based activities and thus to provide appropriate guidance. Assistance Tools for Collaborative Digital Learning Environments An area with similar challenges, where we have sought inspiration from, is that of teacher assistance or awareness and reflection tools on collaborative or open-ended digital learning environments. Similar to practice- based learning, this requires much more than providing simple descriptive statistics of students’ activities. For example, with the aim of supporting students’ meta-cognitive processes in science and mathematics education the METAFORA project developed a bespoke digital platform where students undertake collaborative challenges, describe and enact their plans while working with open-ended environments or games (Dragon et al., 2013). Tracking student activity allows data aggregation and visualization for the teacher in terms of timelines or other charts. Earlier work looked into providing synchronous information to support timely teacher intervention utilising the, familiar by now, traffic light metaphor for showing which students are active, inactive or in need of help in an exploratory digital environment for mathematics (Gutierrez-Santos, Geraniou, Pearce- Lazard, & Poulovassilis, 2012). Roman et al. (2012) explored patterns of collaborative conversation at a non- interactive table aiming to provide information regarding students’ learning process, Martinez-Maldonado et al. (2013) investigated students’ collaborative interactions during their work on an interactive tabletop, Gutierrez- Santos et al. (2012) looked at students’ learning progress and need for help in the context of learning programming and Mercier et al. (2015) studied the collaborative problem solving process within the context of multi-touch technology. Although the aforementioned work points to the potential of tools for increasing awareness, supporting reflection and facilitating decision-making and intervention, to our knowledge, the research presented in this paper is the first attempt to provide information regarding students’ progress during practice-based learning activities. Contextual Inquiry into Practice-based STEM Learning It is by now well understood that design and evaluation of learning analytics tools targeted at teachers (or facilitators in general) and learners, requires techniques and methods from different disciplines, such as software engineering, human- computer interaction and education (Martinez-Maldonado et al., 2015). As discussed in detail by Martinez-Maldonado et al. (2015), while software engineering or human-computer interaction have a lot of methods to offer in relation to establishing technical or usability requirements and for evaluating systems, they may underestimate the learning context. In our previous experience from participatory design, for example, particularly with teachers, the lack of previous experience on tools that can support decision- making makes it really difficult to elicit requirements (Mavrikis, Gutierrez-Santos, Geraniou, Noss, & Poulovassilis, 2013). Instead, in such occasions it is necessary to adopt methodologies that appreciate the need of providing participants the opportunity to directly experience a situation and provide meaningful feedback (Mavrikis et al., 2013). Several methodologies have been used the last few years for designing and evaluating learning analytics tools. One approach that is particularly well suited to our aims is the so-called Learning Awareness Tools User eXperience (LATUX) workflow (Martinez-Maldonado et al., 2015). It was recently put forward as an approach to designing and deploying awareness tools in the classroom by an iterative process of problem definition, low- and higher-fidelity prototypes, pilot studies and validation in-the-wild sessions that can help designers to pay attention to the pedagogical requirements underlying the use of the awareness tools under design. However, even the initial ‘problem identification’ stage requires recognising that in-depth understanding of user behaviour can only be achieved by following a human-centered design process that observes and analyses situations in their actual contexts. This is the main advantage of contextual design approaches or contextual inquiry (Bayer & Holtzblatt, 1998). Hence, in order to understand practice-based learning practices, situate our work in the context of real users and uncover potentials for technology support, we commenced by conducting a contextual inquiry into several STEM learning environments. 41 Method Our contextual inquiry was based on the ethnographic method (Hammersley & Atkinson, 1995), combining participative and observational approaches. We visited ten formal educational institutions in four European countries and interviewed 25 STEM teachers and facilitators. We asked questions about the learning environment, the people, spaces and materials involved in the learning process. Each interview lasted for 1.5 to 2 hours and was digitally audio recorded with participants’ permission. The interviews were conducted face-to- face and were later transcribed verbatim for analysis. Additionally, we conducted a total of nine hours of in-situ observations during STEM classes in the same educational environments. We focused on gaining insights into class dynamics and interaction with learning materials, as well as between peers and teachers within different learning settings. We complemented our data with opportunistic, conversational interviews with a total of 15 students at the end of the observational sessions. Our contextual inquiry was guided by two main research objectives 1) To understand the practices of teachers and learners and their attitude to learning, in the face of material, spatial and logistic constraints and how technological tracing and data analytical augmentation could support them, and subsequently 2) To explore the design of visualizations of practice-based learning activities that can capture aspects of the hands-on, open-ended, collaborative nature of practice-based STEM learning. Thematic analysis was performed, applying an iterative coding scheme with a mix of both deductive and inductive codes (Braun & Clarke, 2006). The resulting coding scheme included learning activities, motivations and attitudes towards tutoring, assessment and the learning process, challenges, as well as socio-material and socio-spatial relationships between users, materials and spaces in the learning process. While the detailed discussion of thick descriptions of the resulting findings is out of the scope of this paper, in the following we present a summarised set of opportunity areas for research and design of technological data-driven augmentations for practice-based learning. Findings from the Contextual Enquiry Study 1) Support Replay and Self-tracking: Hands-on demonstrations are an often-employed teaching strategy, as teachers believe it is necessary as well as stimulating for students to see the correct step-by-step execution of a hands- on activity (e.g. building a circuit) and comprehend and reflect on the steps behind it. This practice also applies to teachers’ in-class tutoring patterns, which often include conducting hands-on mini- demonstrations with individual groups, live- coding in front of the class to highlight specific problems or error patterns, or ‘reverse engineering’ of students’ current outcome in order to find coding or circuitry problems. However, with several individual groups with different levels of knowledge, it is often difficult (or impossible) to trace their mistakes and ‘replay’ the errors. 2) Capture and Visualize Programming / Hands- on Issues: Teachers argue that they often become aware of students’ difficulties during programming and hands-on activities too late, when students are already stuck on larger, more complex issues. They believe they are unable to supervise several student groups simultaneously, and students’ also often lack the motivation and self-regulation skills to identify and report on issues. 3) Promote and Leverage Documentation: According to teachers documentation is increasingly integrated in curricula and assessment criteria. Its implementation during the learning process was found as a challenge, yet it is valued. On the other hand, students find it as tedious and make incomplete, unreflective posts. Nevertheless, they enjoy documentation with digital tools (e.g. Facebook). 4) Support Immediate, Opportunistic Means for Feedback & Documentation: Documenting can be disruptive to learners - especially in hands-on learning environments. Playful, opportunistic mobile documentation could facilitate the process, complemented by a system that tracks and captures important learning events. 5) Support Non-linear Tutoring and Orchestration: Teachers claim that in-class tutoring of hands- on activities is a highly intense and dynamic activity that requires teachers’ attention and engagement at multiple levels. Teachers need to walk around, observe and visit students and attend to their questions and problems, while being able to keep track and give feedback to other students (who sometimes might not even need it). Yet teachers are able to be only at one place at a time, which makes it challenging to attend to specific students’ needs and orchestrate well their feedback. Combining a tracking system that is aware of students’ issues or feedback requests, with on-demand visual feedback through situated, and distributed devices, could provide means to overcome the inevitably ‘sequential’ nature of teachers’ feedback dynamics 42 and allow them to prioritise and orchestrate his tutoring scheme. 6) Multi-purpose spaces & dynamics: Students often use school spaces for multiple purposes – such as the workshop for brainstorming rather than just product work. Tracing their presence in these various spaces might yield information about their project development paths. 7) Capture and Visualize Collaboration: Teachers believe that collaboration is an important process for learning and an effective way to expand and reinforce one’s knowledge. They try to develop a positive attitude in students towards cooperation with others by organising teamwork activities. Collaboration is assessed after continuous observation of teamwork and teachers usually keep track of it through personal observation notes that add to the overall ‘assessment’ profile of the learner at the end of the course. However, as teachers point out this assessment strategy is highly subjective and difficult to track and thus, it remains challenging to capture collaborative skills effectively. Then these findings were mapped to the design features of our visualisations as presented in figure 1 below. Figure 1. Mapping the contextual enquiry findings with design features Prototyping Visualisations After two prototyping iterations, we developed the visualisation presented in Figure 2. It corresponds to our findings from our contextual enquiry study in practice-based learning environments. Figure 2. Snapshot of visualization designed using taking into account findings of the contextual enquiry study 43 As can be seen, it includes labels for each component of the visualisation: A. Visualization of Physical Computing Kit Activity: Using an Arduino-based Smart Learning Kit, we were able to visualize the hardware and software components in use and time spend using them. For example, the “BTN” represents the use of the button component by the students, making a physical computing project. They clearly use it throughout the whole working session. Yet they use the “ACR” (accelerometer) much less frequently - showing project development patterns. B. Figure 3. Physical computing tools’ presentation in the visualisation We chose to represent the physical connection of a component as a strong thin line, the software use as a rectangle, each extending for the period of time for which they were either physically or digitally connected (Figure 3). The color of the component’s visual representation depends on whether it was an input (button, sensor, etc.) or output - thus aligning with the physical elements’ own placards as well. Any connection made is represented as a triangle on the element connected and each end of the connection on that element is represented as a square at the moment of the disconnection, again placed in line with that element’s general linear representation track. C. Sentiment Feedback Visualisation: We visualized the button presses from the Sentiment Feedback Buttons (designed as part of the prototype) with a lightbulb icon (positive sentiment, e.g. “eureka idea”) and a storm-cloud icon (negative sentiment, e.g. “frustrated”, “stuck”). The icons were displayed over the visualization timeline at the moment of a corresponding button press. D. Screenshot From the Workstation and the Computer Screen: We implemented a snapshot ability into each of the Sentiment Buttons such that when pressed, an overview camera from the workstation is triggered to take a picture of the students’ working environment. At the same time, a button press triggers the system to take a screenshot from the computer screen. Snapshots expand upon mouse hover and swap upon click to show the other image associated with this same time. E. Overall Timeline with a Manipulatable Interface: A student or teacher can choose a slice of time that is as small as one minute or expand the slice to the full length of the session. They can look at the minute of a ‘frustration’ button press and see what modules were in use. They can also zoom out and look at the data patterns over the full period of project development. This view reveals patterns of usage behaviour such as the progression in complexity. We are at the stage of evaluating our visualisation in real world teaching environments. We are interested to find out how educators and students engage with learning visualizations of data originating from their practice-based work, in particular supporting students’ reflections, discussions, and self- regulation, as well as educators’ awareness and assessment of the learning process Our initial feedback from teachers and students demonstrate that the visualization could support valuable processes within practice- based STEM learning and teaching. Some of the most salient are students’ collective post - reflection and debriefing of specific difficulties within a project, and the facilitation of communication on those issues in the group and with their teacher. However, we would like to evaluate the visualisation in formal and informal teaching environments using robust research criteria with bigger samples in order to be able to draw better conclusions regarding its potential use in classrooms. 44 Conclusions and implications In this paper, we presented our human-centered process for generating visualisations of face-to-face, practice- based learning activities, based on a contextual inquiry study of real world settings. We believe as our colleagues (Yu & Nakamura, 2010) that technology can capture only certain aspects of student interactions during such rich learning situations as practice-based learning activities. Hence, it is challenging to present the practice-based learning process as a whole. However, our visualisation reflects some aspects of the learning process that are considered as important by teachers and, as our initial feedback sessions demonstrate, this approach can be valuable for providing support to both teachers and students. Yet, there are other elements, which we identified in our studies as relevant and encourage future investigations, such as capturing and visualization of more heterogenous types of activities (e.g. sketching), or surfacing collaboration patterns. We hope that our visualisation will generate a productive discussion at the workshop and we can get some feedback on our visualisation of practice-based learning process as well as our approach to design it. Acknowledgments This work was funded by the PELARS project (GA No. 619738) under the Seventh Framework Programme of the European Commission. References Bayer, H., & Holtzblatt, K. (1998). Contextual Design: Defining Customer-Centred Systems. San Francisco: Morgan Kaufmann. Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. Clark, R. E. (2009). How much and what type of guidance is optimal for learning from instruction? In S. Tobias & T. M. Duffy (Eds.), Constructivist theory applied to instruction: Success of failure? (pp. 158-183). New York: Routledge, Taylor and Francis. Coffrin, C., Corrin, L., de Barba, P., & Kennedy, G. (2014). Visualizing patterns of student engagement and performance in MOOCs. Paper presented at the The fourth international conference on learning analytics and knowledge. Cukurova, M., & Bennett, J. (2014). An investigation of the effects of a novel teaching approach on students’ learning of chemical ideas. Paper presented at the ESERA 2013 Conference: Science Education Research For Evidence-based Teaching and Coherence in Learning, Nicosia, Cyprus. Dillenbourg, P., Zufferey, G., Alavi, H., Jermann, P., Do-Lenh, S., & Bonnard, Q. (2011). Classroom orchestration: The third circle of usability. Paper presented at the International Conference on Computer Supported Collaborative Learning 2011 (CSCL 2011), Hong Kong. Dragon, T., Mavrikis, M., McLaren, B. M., Harrer, A., Kynigos, C., Wegerif, R., & Yang, Y. (2013). Metafora: A web-based platform for learning to learn together in science and mathematics. Learning Technologies, IEEE Transactions on, 6(3), 197-207. Govaerts, S., Verbert, K., Klerkx, J., & Duval, E. (2010). Visualizing activities for self-reflection and awareness. ICWL, 91-100. Gutierrez-Santos, S., Geraniou, E., Pearce-Lazard, D., & Poulovassilis, A. (2012). Desing of teacher assistance tools in an exploratory learning environment for algebraic generalization. Learning Technologies, IEEE Transactions on, 5(4), 366-376. Hammersley, M., & Atkinson, P. (1995). Ethnography: Principles in practice. London: Routledge. Lafford, B. A. (2004). Review of Tell me More Spanish. Journal on Language Learning and Technology, 8(3), 21-34. Martinez-Maldonado, R., Dimitriadis, Y., Martinez-Monés, A., Kay, J., & Yacef, K. (2013). Capturing and analyzing verbal and pyhsical collaborative learning interactions at an enriched interactive tabletop. Computer-Supported Collaborative Learning, 8, 455-485. Martinez-Maldonado, R., Pardo, A., Mirriahi, N., Yacef, K., Kay, J., & Clayphan, A. (2015). The LATUX workflow: Designing and deploying awareness tools in technology-enabled learning settings. Paper presented at the Fifth International Conference on Learning Analytics and Knowledge. Mavrikis, M., Gutierrez-Santos, S., Geraniou, E., Noss, R., & Poulovassilis, A. (2013). Iterative context engineering to inform the design of intelligent exploratory learning environments for the classroom. In R. Luckin, S. Puntambekar, P. Goodyear, B. L. Grabowski, J. Underwood, & N. Winters (Eds.), Handbook of Design in Educational Technology (pp. 80-92). London: Routledge. 45 Mercier, E., Vourloumi, G., & Higgins, S. (2015). Student interactions and the developme nt of ideas in multi- touch and paper-based collaborative mathematical problem solving. British Journal of Educational Technology. doi:10.1111/bjet.12351 Millar, R. (2004). The role of practical work in the teaching and learning of science Retrieved from Washington, DC.: Race, P. (2001). A briefing on self, peer and group assessement. York: Higher Education Academy. Roman, F., Mastrogiacomo, S., Mlotkowski, D., Kaplan, F., & Dillenbourg, P. (2012). Can a table regulate participation in top level managers' meetings? Paper presented at the 17th International Conference on Supporting Group Work, New York. Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2014). Supporting teachers in guiding collaborating students: Effects of learning analytics in CSCL. Computers & Education, 79, 28- 39. Verbert, K., Govaerts, S., Duval, E., Santos, J. L., van Assche, F., Parra, G., & Klerkx, J. (2014). Learning dasboards: an overview and future research opportunities. Personal and Ubiquitous Computing, 18(6), 1499-1514. Yu, Z., & Nakamura, Y. (2010). Smart Meeting Systems: A Survey of State-of-the-Art and Open Issues. ACM Computing Surveys, 42(2), 1-20. Zhang, D., Zhao, J. L., Zhou, L., & Nunamaker, J. F. (2004). Can e-learning replace classroom learning? Communications of the ACM, 47(5), 75-81 46