This paper describes a multi-modal dataset collected for studying collaborative, engineering design cognition among undergraduate students. Students worked in pairs to solve two engineering design challenges, and also participated in a variety of interventions aimed to improve the quality of the learning experience. While students completed these hands-on tasks, multimodal data was captured using Xbox Kinect, Leap motion, a high definition web camera, and Affectiva Q-sensor.
Multimodal data capture capabilities and multimodal learning analytics [1]–[3] are rapidly growing. Research and practitioners now have the opportunity to collect a wealth of multimodal process data as students complete collaborative tasks in the physical and/or digital world. What’s more, these tools may offer a different perspective into students’ learning experiences. For example, sensors like the Xbox Kinect and high definition web-cameras have the ability to store rich, high frequency data about how learners are physically engaging with a given task. Moreover, different bio-physiological sensors (e.g., Empatica E4 or the Affectiva Q-sensor) have the ability to capture data that may be too fine-grained and minute for a human to detect.
In this paper I describe a dataset that was recently collected to study engineering design
cognition as perceived through a host of multimodal sensors. However, before I delve
into describing the data, I will briefly provide the motivation for the collecting this data.
I will then move into describing the different pieces of data collected and some of the
challenges collecting the data. I will conclude with some of the on-going work being
done to clean the dataset and prepare it for dissemination.
Over the past few years there has been growing interest in improving K-16 engineering
education [4]–[7]. However, in the same way that Problem-Based Learning faced many
challenges, engineering design, and the Maker Movement more broadly, are also
susceptible to the phenomena of “doing without understanding” [
Study Participants 54 students from a West Coast community college, who ranged in terms of age and prior experience with engineering design, participated in this study. The students included several majors and students with different career goals. The participants received course credit for their involvement, and in this way were a sample of convenience. 3.2
The study uses a 2-by-2 design where students worked in pairs on two engineering design tasks and participated in two interventions. Student pairing was based on availability. Students also completed other activities to allow us to better understand and analyze their learning experience. A diagrammatic representation of the events is shown in Figure 1. The total experience lasted about one hour per group. This amount of total time mirrors a number of engineering design oriented “maker” challenges that take place at after-school programs in libraries and museums around the country. A detailed description of the study is included in the following paragraphs.
to draw either one design (three minutes) or three very different designs (one minute per design). We will refer to these conditions as Single and Multiple.
Task #1: Paper and Textbook Task. Students were asked to use one sheet of printer paper to construct a structure that could support one or more engineering textbooks at least three inches off the table (see Figures 2-4 for examples). Students had six minutes to complete this task. The task aimed to help students realize that the configuration of the piece of paper was of significant import, and that the material was not the only factor. The number of books that each pair’s design could support was recorded. Question for an Expert. After the first task, students wrote down questions that they would ask an expert in mechanical engineering or engineering design. Students had approximately three minutes to write down their questions.
Intervention #2: Video or Discussion. Pairs of students were randomly assigned to watch a short informational video while others participated in a discussion about how the first task had proceeded.
Task #2: Paper Plate and Beans Task. Students had 10 minutes to use one paper plate, two feet of tape, three wooden sticks and four straws. These materials were used to build a structure that could support a mass of 0.5 lb. as high off the table as possible (see Figures 8 & 9 for examples). This task looked to build on similar principles as the first task, but with greater variability in the materials, and with the added goal of height optimization. Note: Both Task #1 and Task #2 sit at the fuzzy intersection of engineering and making which is currently be advanced by a number of schools, museums, government organizations, etc. [7], [13]. The height of each pair’s design was recorded. Pre-, Mid- and Post-tests. Students were asked to identify principles or mechanisms in three example structures (see Figures 5-7) as pre-, mid- and post-tests (note: they weren’t framed as tests, but as ways to help the students do better on the tasks). Students were given approximately 1 minute to write down their ideas for why each structure was stable. Students had access to their prior responses and were allowed to copy or update their previous ideas. Responses to these questions served as the basis for documenting conceptual change.
Following the post test, students also answered a number of questions about their experience, and completed two transfer tasks that asked them to compare different designs, and to describe how they would teach a younger student how to go about completing a similar engineering design task.
Pre-test Intervention #1 Paper & Textbook
Questions for
Expert Intervention #2
Mid-test Paper Plate & Beans
Post-test
EDA Stress Tests
In addition to the time-stamped task annotations and hand written artifacts collected for this study, I was also able to collect data from a number of multimodal sensors. These sensors include: 1 high definition web camera (audio/video data), Xbox Kinect (multichannel audio, skeletal tracking and frontal images), Affectiva Q-sensor (electro-dermal activation, and hand/wrist movement), three Leap motion controllers (3-axis hand/wrist and prop movement). The high definition web camera was positioned directly over top of the students and collected data at 30 frames per second (Figure 10). The Xbox Kinect was positioned approximately 1.5 meters in front of the students and collected skeletal tracking data at 10 frames per second, and frontal images (Figure 11) at one frame per second. The audio from the Xbox Kinect included all four channels and was captured at 16kHz. The Affectiva Q-sensor was worn on the wrist, and collected data at a rate of 8 samples per second. Furthermore, two stress tests were administered at the conclusion of the task in order to provide an individualized baseline for each student under stress. Finally, a Leap motion sensor was positioned to the side of each participant, and over the top (next to the web camera). Leap data was captured at approximately 60 Hz.
The current dataset includes all of the raw data described above, in addition to several pieces of data that were extracted from the different modalities. For example, transcripts are available for all participants during both tasks. For Task #2 the transcripts are timestamped, and have been aligned with the audio to simplify prosodic analysis, for example. Additionally, the frontal images were used to provide second by second head pose estimation and automatic facial expression analysis[14], [15]. In particular, I have an estimate for the direction of each user’s gaze, evidence of facial action units, and evidence of basic facial expressions. Finally, demographic information about each student, their performance in school, high school grade point average, etc. is also available. 4
A particular challenge in collecting this data set was synchronizing the data across the different machines being used to collected the different modalities. Part of this process was simplified by running synchronization tasks before several of experiments, but considerable effort was still required to properly synchronize data from the different modalities. Additionally, providing accurate real-time event annotation was a challenge (though in the end this greatly eased the synchronization process). Another challenge encountered was the sporadic nature of some of the data collection tools. For example, overhead video, audio, and/or frontal images is missing from a number of pairs. This lost data is largely the result of software failing to operate as expected.
Similar challenges exist in analyzing and visualizing the current data in such a way that is meaningful. Many of the analytic tools available cater towards working in a particular modality, but there seem to be few tools that can effectively be used with multiple modalities, outside of custom scripts in MATLAB and/or Python.