Practice-based learning activities are an important aspect of education, particularly for science, technology, engineering and mathematics (STEM) subjects. Their immense importance to STEM curricula is unequivocal and so are the teachers' and students' need for support during those activities. However, considering the open-ended and hands-on nature of practice-based learning activities, designing, deploying and validating learning analytics visualisations remains as a significant challenge for the state of-the-art learning analytics. In this paper, we present our human- centered contextual enquiry approach for generating requirements and its preliminary results in the form of visualisations that have the potential to support facilitators and learners. Although there have been certain attempts to provide learning analytics for increasing awareness, supporting reflection and facilitating decision-making and intervention, to our knowledge, our research presented is the first attempt to provide such information regarding students' progress during practice-based learning activities.
A number of visualisation tools have been developed for online, face-to-face, and blended learning
settings, where this data is more readily available. Most of these attempts aim to support teachers (Verbert
et al., 2014), but some applications have also been developed to support students’ awareness and
selfreflection (e.g.
These solutions, however, do not necessarily transfer in open- ended, practice-based learning where the technical challenges are very different and the usability and pedagogical requirements are not yet well understood. First, practice-based learning activities usually take place simultaneously in multiple groups of students, sometimes in a range of physical spaces and across a large time- span. In addition, the diverse set of digital and non-digital activities cannot always be tracked keeping practice-based learning largely out of the scope of current learning analytics trends, despite its immense importance to STEM curricula. We are interested in investigating whether learning analytics can support the challenging role of the teacher or facilitator in such settings and/or help students reflect on their own practice. The challenges teachers face during practice-based learning, particularly in formal education, are well documented. Teachers are rarely aware of the processes followed by students during these activities (Race, 2001), and it is challenging for them to provide appropriate support to individual students, who have different needs, strengths and weaknesses (Zhang, Zhao, Zhou, & Nunamaker, 2004).
Teachers can only be aware of what a small number of students are doing at any one time in the classroom. It is, therefore, hard for teachers to know which students are making progress, and which are in difficulty and in need of additional support. It is a challenge for teachers to understand the process by which students have arrived at the current state of their practice- based activities and thus to provide appropriate guidance.
An area with similar challenges, where we have sought inspiration from, is that of teacher assistance or
awareness and reflection tools on collaborative or open-ended digital learning environments. Similar to
practice- based learning, this requires much more than providing simple descriptive statistics of students’
activities. For example, with the aim of supporting students’ meta-cognitive processes in science and
mathematics education the METAFORA project developed a bespoke digital platform where students
undertake collaborative challenges, describe and enact their plans while working with open-ended
environments or games
It is by now well understood that design and evaluation of learning analytics tools targeted at teachers (or
facilitators in general) and learners, requires techniques and methods from different disciplines, such as
software engineering, human- computer interaction and education
Several methodologies have been used the last few years for designing and evaluating learning
analytics tools. One approach that is particularly well suited to our aims is the so-called Learning
Awareness Tools User eXperience (LATUX) workflow
Our contextual inquiry was based on the ethnographic method
We focused on gaining insights into class dynamics and interaction with learning materials, as well as between peers and teachers within different learning settings. We complemented our data with opportunistic, conversational interviews with a total of 15 students at the end of the observational sessions. Our contextual inquiry was guided by two main research objectives 1) To understand the practices of teachers and learners and their attitude to learning, in the face of material, spatial and logistic constraints and how technological tracing and data analytical augmentation could support them, and subsequently 2) To explore the design of visualizations of practice-based learning activities that can capture aspects of the hands-on, open-ended, collaborative nature of practice-based STEM learning.
Thematic analysis was performed, applying an iterative coding scheme with a mix of both
deductive and inductive codes
Findings from the Contextual Enquiry Study 1) Support Replay and Self-tracking: Hands-on demonstrations are an often-employed teaching strategy, as teachers believe it is necessary as well as stimulating for students to see the correct step-by-step execution of a hands- on activity (e.g. building a circuit) and comprehend and reflect on the steps behind it. This practice also applies to teachers’ in-class tutoring patterns, which often include conducting hands-on mini- demonstrations with individual groups, livecoding in front of the class to highlight specific problems or error patterns, or ‘reverse engineering’ of students’ current outcome in order to find coding or circuitry problems. However, with several individual groups with different levels of knowledge, it is often difficult (or impossible) to trace their mistakes and ‘replay’ the errors. 2) Capture and Visualize Programming / Hands- on Issues: Teachers argue that they often become aware of students’ difficulties during programming and hands-on activities too late, when students are already stuck on larger, more complex issues. They believe they are unable to supervise several student groups simultaneously, and students’ also often lack the motivation and self-regulation skills to identify and report on issues. 3) Promote and Leverage Documentation: According to teachers documentation is increasingly integrated in curricula and assessment criteria. Its implementation during the learning process was found as a challenge, yet it is valued. On the other hand, students find it as tedious and make incomplete, unreflective posts. Nevertheless, they enjoy documentation with digital tools (e.g. Facebook). 4) Support Immediate, Opportunistic Means for Feedback & Documentation: Documenting can be disruptive to learners - especially in hands-on learning environments. Playful, opportunistic mobile documentation could facilitate the process, complemented by a system that tracks and captures important learning events. 5) Support Non-linear Tutoring and Orchestration: Teachers claim that in-class tutoring of handson activities is a highly intense and dynamic activity that requires teachers’ attention and engagement at multiple levels. Teachers need to walk around, observe and visit students and attend to their questions and problems, while being able to keep track and give feedback to other students (who sometimes might not even need it). Yet teachers are able to be only at one place at a time, which makes it challenging to attend to specific students’ needs and orchestrate well their feedback. Combining a tracking system that is aware of students’ issues or f eedback requests, with on-demand visual feedback through situated, and distributed devices, could provide means to overcome the inevitably ‘sequential’ nature of teachers’ feedback dynamics and allow them to prioritise and orchestrate his tutoring scheme. 6) Multi-purpose spaces & dynamics: Students often use school spaces for multiple purposes – such as the workshop for brainstorming rather than just product work. Tracing their presence in these various spaces might yield information about their project development paths. 7) Capture and Visualize Collaboration: Teachers believe that collaboration is an important process for learning and an effective way to expand and reinforce one’s knowledge. They try to develop a positive attitude in students towards cooperation with others by organising teamwork activities. Collaboration is assessed after continuous observation of teamwork and teachers usually keep track of it through personal observation notes that add to the overall ‘assessment’ profile of the learner at the end of the course. However, as teachers point out this assessment strategy is highly subjective and difficult to track and thus, it remains challenging to capture collaborative skills effectively.
Then these findings were mapped to the design features of our visualisations as presented in figure 1 below.
After two prototyping iterations, we developed the visualisation presented in Figure 2. It corresponds to our findings from our contextual enquiry study in practice-based learning environments. As can be seen, it includes labels for each component of the visualisation: A. Visualization of Physical Computing Kit Activity: Using an Arduino-based Smart Learning Kit, we were able to visualize the hardware and software components in use and time spend using them. For example, the “BTN” represents the use of the button component by the students, making a physical computing project. They clearly use it throughout the whole working session. Yet they use the “ACR” (accelerometer) much less frequently - showing project development patterns. We chose to represent the physical connection of a component as a strong thin line, the software use as a rectangle, each extending for the period of time for which they were either physically or digi tally connected (Figure 3). The color of the component’s visual representation depends on whether it was an input (button, sensor, etc.) or output - thus aligning with the physical elements’ own placards as well. Any connection made is represented as a triangle on the element connected and each end of the connection on that element is represented as a square at the moment of the disconnection, again placed in line with that element’s general linear representation track. C. Sentiment Feedback Visualisation: We visualized the button presses from the Sentiment Feedback Buttons (designed as part of the prototype) with a lightbulb icon (positive sentiment, e.g. “eureka idea”) and a storm-cloud icon (negative sentiment, e.g. “frustrated”, “stuck”). The icons were displayed over the visualization timeline at the moment of a corresponding button press. D. Screenshot From the Workstation and the Computer Screen: We implemented a snapshot ability into each of the Sentiment Buttons such that when pressed, an overview camera from the workstation is triggered to take a picture of the students’ working environment. At the same time, a button press triggers the system to take a screenshot from the computer screen. Snapshots expand upon mouse hover and swap upon click to show the other image associated with this same time.
E. Overall Timeline with a Manipulatable Interface: A student or teacher can choose a slice of time that is as small as one minute or expand the slice to the full length of the session. T hey can look at the minute of a ‘frustration’ button press and see what modules were in use. They can also zoom out and look at the data patterns over the full period of project development. This view reveals patterns of usage behaviour such as the progression in complexity.
We are at the stage of evaluating our visualisation in real world teaching environments. We are interested to find out how educators and students engage with learning visualizations of data originating from their practice-based work, in particular supporting students’ reflections, discussions, and selfregulation, as well as educators’ awareness and assessment of the learning process Our initial feedback from teachers and students demonstrate that the visualization could support valuable processes within practice- based STEM learning and teaching. Some of the most salient are students’ collective post reflection and debriefing of specific difficulties within a project, and the facilitation of communication on those issues in the group and with their teacher. However, we would like to evaluate the visualisation in formal and informal teaching environments using robust research criteria with bigger samples in order to be able to draw better conclusions regarding its potential use in classrooms.
In this paper, we presented our human-centered process for generating visualisations of face-to-face, practice- based learning activities, based on a contextual inquiry study of real world settings. We believe as our colleagues (Yu & Nakamura, 2010) that technology can capture only certain aspects of student interactions during such rich learning situations as practice-based learning activities. Hence, it is challenging to present the practice-based learning process as a whole. However, our visualisation reflects some aspects of the learning process that are considered as important by teachers and, as our initial feedback sessions demonstrate, this approach can be valuable for providing support to both teachers and students. Yet, there are other elements, which we identified in our studies as relevant and encourage future investigations, such as capturing and visualization of more heterogenous types of activities (e.g. sketching), or surfacing collaboration patterns. We hope that our visualisation will generate a productive discussion at the workshop and we can get some feedback on our visualisation of practice -based learning process as well as our approach to design it.
This work was funded by the PELARS project (GA No. 619738) under the Seventh Framework Programme of the European Commission. Mercier, E., Vourloumi, G., & Higgins, S. (2015). Student interactions and the developme nt of ideas in multi- touch and paper-based collaborative mathematical problem solving. British Journal of Educational Technology. doi:10.1111/bjet.12351 Millar, R. (2004). The role of practical work in the teaching and learning of science Retrieved from
Washington, DC.: Race, P. (2001). A briefing on self, peer and group assessement. York: Higher Education Academy. Roman, F., Mastrogiacomo, S., Mlotkowski, D., Kaplan, F., & Dillenbourg, P. (2012). Can a table regulate participation in top level managers' meetings? Paper presented at the 17th International Conference on Supporting Group Work, New York.
Van Leeuwen, A., Janssen, J., Erkens, G., & Brekelmans, M. (2014). Supporting teachers in guiding collaborating students: Effects of learning analytics in CSCL. Computers & Education, 79, 2839.
Verbert, K., Govaerts, S., Duval, E., Santos, J. L., van Assche, F., Parra, G., & Klerkx, J. (2014).
Learning dasboards: an overview and future research opportunities. Personal and Ubiquitous Computing, 18(6), 1499-1514.
Yu, Z., & Nakamura, Y. (2010). Smart Meeting Systems: A Survey of State-of-the-Art and Open Issues.
ACM Computing Surveys, 42(2), 1-20.
Zhang, D., Zhao, J. L., Zhou, L., & Nunamaker, J. F. (2004). Can e-learning replace classroom learning? Communications of the ACM, 47(5), 75-81