Engineering Design with Everyday Materials Multi- modal Dataset Marcelo Worsley1 1 Northwestern University, Evanston, IL 60208 marcelo.worsley@northwestern.edu Abstract. This paper describes a multi-modal dataset collected for studying col- laborative, engineering design cognition among undergraduate students. Students worked in pairs to solve two engineering design challenges, and also participated in a variety of interventions aimed to improve the quality of the learning experi- ence. While students completed these hands-on tasks, multimodal data was cap- tured using Xbox Kinect, Leap motion, a high definition web camera, and Affec- tiva Q-sensor. Keywords: Multimodal learning analytics, engineering design cognition, col- laboration 1 Introduction Multimodal data capture capabilities and multimodal learning analytics [1]–[3] are rap- idly growing. Research and practitioners now have the opportunity to collect a wealth of multimodal process data as students complete collaborative tasks in the physical and/or digital world. What’s more, these tools may offer a different perspective into students’ learning experiences. For example, sensors like the Xbox Kinect and high definition web-cameras have the ability to store rich, high frequency data about how learners are physically engaging with a given task. Moreover, different bio-physiolog- ical sensors (e.g., Empatica E4 or the Affectiva Q-sensor) have the ability to capture data that may be too fine-grained and minute for a human to detect. In this paper I describe a dataset that was recently collected to study engineering design cognition as perceived through a host of multimodal sensors. However, before I delve into describing the data, I will briefly provide the motivation for the collecting this data. I will then move into describing the different pieces of data collected and some of the challenges collecting the data. I will conclude with some of the on-going work being done to clean the dataset and prepare it for dissemination. 2 Dataset Motivation Over the past few years there has been growing interest in improving K-16 engineering education [4]–[7]. However, in the same way that Problem-Based Learning faced many 2 challenges, engineering design, and the Maker Movement more broadly, are also sus- ceptible to the phenomena of “doing without understanding” [8]–[10]. Hence, a primary motivation for collecting this dataset is to study and compare different strategies for promoting learning in the context of an open-ended, engineering design task. In partic- ular, we examine how different interventions can improve the quality of an engineering design, or Making, experience, and the ways that this is evidenced through multimodal data, in accordance with prior work [11], [12]. 3 Methodology 3.1 Study Participants 54 students from a West Coast community college, who ranged in terms of age and prior experience with engineering design, participated in this study. The students in- cluded several majors and students with different career goals. The participants re- ceived course credit for their involvement, and in this way were a sample of conven- ience. 3.2 Study Description The study uses a 2-by-2 design where students worked in pairs on two engineering design tasks and participated in two interventions. Student pairing was based on avail- ability. Students also completed other activities to allow us to better understand and analyze their learning experience. A diagrammatic representation of the events is shown in Figure 1. The total experience lasted about one hour per group. This amount of total time mirrors a number of engineering design oriented “maker” challenges that take place at after-school programs in libraries and museums around the country. A detailed description of the study is included in the following paragraphs. Intervention #1: Single Design vs. Multiple Designs. Pairs were randomly assigned to draw either one design (three minutes) or three very different designs (one minute per design). We will refer to these conditions as Single and Multiple. Task #1: Paper and Textbook Task. Students were asked to use one sheet of printer paper to construct a structure that could support one or more engineering textbooks at least three inches off the table (see Figures 2-4 for examples). Students had six minutes to complete this task. The task aimed to help students realize that the configuration of the piece of paper was of significant import, and that the material was not the only factor. The number of books that each pair’s design could support was recorded. 3 Question for an Expert. After the first task, students wrote down questions that they would ask an expert in mechanical engineering or engineering design. Students had approximately three minutes to write down their questions. Intervention #2: Video or Discussion. Pairs of students were randomly assigned to watch a short informational video while others participated in a discussion about how the first task had proceeded. Task #2: Paper Plate and Beans Task. Students had 10 minutes to use one paper plate, two feet of tape, three wooden sticks and four straws. These materials were used to build a structure that could support a mass of 0.5 lb. as high off the table as possible (see Figures 8 & 9 for examples). This task looked to build on similar principles as the first task, but with greater variability in the materials, and with the added goal of height optimization. Note: Both Task #1 and Task #2 sit at the fuzzy intersection of engineer- ing and making which is currently be advanced by a number of schools, museums, government organizations, etc. [7], [13]. The height of each pair’s design was recorded. Pre-, Mid- and Post-tests. Students were asked to identify principles or mechanisms in three example structures (see Figures 5-7) as pre-, mid- and post-tests (note: they weren’t framed as tests, but as ways to help the students do better on the tasks). Students were given approximately 1 minute to write down their ideas for why each structure was stable. Students had access to their prior responses and were allowed to copy or update their previous ideas. Responses to these questions served as the basis for docu- menting conceptual change. Following the post test, students also answered a number of questions about their experience, and completed two transfer tasks that asked them to compare different de- signs, and to describe how they would teach a younger student how to go about com- pleting a similar engineering design task. 4 Pre-test Intervention #1 Paper & Textbook Questions for Expert Intervention #2 Mid-test Paper Plate & Beans Post-test EDA Stress Tests Figure 1 Sequence of Activities Figure 2 Figure 3 Figure 4 Task 1 Design Task 1 Design Task 1 Design Figure 5 Figure 6 Figure 7 Ladder Image Bridge Image Igloo Image Figure 8 Figure 9 Task 2 Design Task 2 Design 5 3.3 Multimodal Data In addition to the time-stamped task annotations and hand written artifacts collected for this study, I was also able to collect data from a number of multimodal sensors. These sensors include: 1 high definition web camera (audio/video data), Xbox Kinect (multi- channel audio, skeletal tracking and frontal images), Affectiva Q-sensor (electro-der- mal activation, and hand/wrist movement), three Leap motion controllers (3-axis hand/wrist and prop movement). The high definition web camera was positioned di- rectly over top of the students and collected data at 30 frames per second (Figure 10). The Xbox Kinect was positioned approximately 1.5 meters in front of the students and collected skeletal tracking data at 10 frames per second, and frontal images (Figure 11) at one frame per second. The audio from the Xbox Kinect included all four channels and was captured at 16kHz. The Affectiva Q-sensor was worn on the wrist, and col- lected data at a rate of 8 samples per second. Furthermore, two stress tests were admin- istered at the conclusion of the task in order to provide an individualized baseline for each student under stress. Finally, a Leap motion sensor was positioned to the side of each participant, and over the top (next to the web camera). Leap data was captured at approximately 60 Hz. Figure 10 Example Overhead Web Camera Image Figure 11 Example Kinect Frontal Image 6 3.4 Multimodal Data Extraction The current dataset includes all of the raw data described above, in addition to several pieces of data that were extracted from the different modalities. For example, transcripts are available for all participants during both tasks. For Task #2 the transcripts are time- stamped, and have been aligned with the audio to simplify prosodic analysis, for exam- ple. Additionally, the frontal images were used to provide second by second head pose estimation and automatic facial expression analysis[14], [15]. In particular, I have an estimate for the direction of each user’s gaze, evidence of facial action units, and evi- dence of basic facial expressions. Finally, demographic information about each student, their performance in school, high school grade point average, etc. is also available. 4 Challenges A particular challenge in collecting this data set was synchronizing the data across the different machines being used to collected the different modalities. Part of this process was simplified by running synchronization tasks before several of experiments, but considerable effort was still required to properly synchronize data from the different modalities. Additionally, providing accurate real-time event annotation was a challenge (though in the end this greatly eased the synchronization process). Another challenge encountered was the sporadic nature of some of the data collection tools. For example, overhead video, audio, and/or frontal images is missing from a number of pairs. This lost data is largely the result of software failing to operate as expected. Similar challenges exist in analyzing and visualizing the current data in such a way that is meaningful. Many of the analytic tools available cater towards working in a par- ticular modality, but there seem to be few tools that can effectively be used with multi- ple modalities, outside of custom scripts in MATLAB and/or Python. References 1. Blikstein, P. & Worsley, M. Multimodal Learning Analytics: a methodological framework for research in constructivist learning. Journal of. Learning Analytics: 3, 220-238 (2016). 2. Ochoa, X. & Worsley, M. Augmenting Learning Analytics with Multimodal Sensory Data. Journal of. Learning Analytics. 3, 213–219 (2016). 3. Worsley, M. Multimodal learning analytics: enabling the future of learning through multimodal data analysis and interfaces. in Proceedings of the 14th ACM international conference on Multimodal interaction 353–356 (2012). 4. National Research Council. A Framework for K-12 Science Education: Practices, Crosscutting Concepts, and Core Ideas. (National Academies Press, 2012). 5. Next Generation Science Standards: For States, By States. (2013). 6. Martin, L. The promise of the Maker Movement for education. J. Pre-College 7 Eng. Educ. Res. 5, 4 (2015). 7. Honey, M. & Kanter, D. E. Design, Make, Play: Growing the Next Generation of STEM Innovators. (Routledge, 2013). 8. Kolodner, J. L. et al. Journal of the Learning Problem-Based Learning Meets Case-Based Reasoning in the Middle-School Science Classroom : Putting Learning by Design ( tm ) Into Practice. 37–41 (2009). doi:10.1207/S15327809JLS1204 9. Schwartz, D. L. Constructivism in an age of non-constructivist assessments. (1992). 10. Barron, B. et al. Doing With Understanding: Lessons From Research on Problem and Project-Based Learning. J. Learn. Sci. 7, 271–311 (1998). 11. Worsley, M. & Blikstein, P. Leveraging Multimodal Learning Analytics to Differentiate Student Learning Strategies. in Proceedings of the Fifth International Conference on Learning Analytics And Knowledge 360–367 (ACM, 2015). doi:10.1145/2723576.2723624 12. Worsley, M. & Blikstein, P. Using learning analytics to study cognitive disequilibrium in a complex learning environment. Proc. Fifth Int. Conf. Learn. Anal. Knowl. - LAK ’15 426–427 (2015). doi:10.1145/2723576.2723659 13. Vossoughi, S. & Bevan, B. Making and tinkering: A review of the literature. National Research Council Committee on Out of School Time STEM. (2014). 14. Baltrušaitis, T., Robinson, P. & Morency, L.-P. OpenFace: an open source facial behavior analysis toolkit. in IEEE Winter Conference on Applications of Computer Vision (2016). 15. Bartlett, M., Littlewort, G., Wu, T. & Movellan, J. Computer expression recognition toolbox. in Automatic Face & Gesture Recognition, 2008. FG’08. 8th IEEE International Conference on 1–2 (2008).