Educational research has demonstrated the importance of embodiment in the design of student learning environments, connecting bodily actions to critical concepts. Gestural recognition algorithms have become important tools in leveraging this connection but are limited in their development, focusing primarily on traditional machine-learning paradigms. We describe our approach to real-time learning analytics, using a gesturerecognition system to interpret movement in an educational context. We train a hybrid parametric, hierarchical hidden-Markov model using a one-shot construct, learning from singular, user-defined gestures. This model gives us access to three different modes of data streams: skeleton positions, kinematics features, and internal model parameters. Such a structure presents many challenges including anticipating the optimal feature sets to analyze and creating effective mapping schemas. Despite these challenges, our method allows users to facilitate productive simulation interactions, fusing of these streams into embodied semiotic structures defined by the individual. This work has important implications for the future of multimodal learning analytics and educational technology.
The connection between embodiment and human cognition has become increasingly
established within an array of academic domains [
While the opportunity to create more advanced models that engage MMLA for
embodied learning exists, the majority of interactive learning systems employ
directmanipulation paradigms (i.e. pushing virtual buttons, grasping virtual objects) rather
than empowering learners to use expressive gestures based on their own embodied
intuitions [
Systems which utilize gesture-based input seem like a natural way to strengthen the connection between body and mind in digital interaction yet many challenges remain for the user and designer of such systems. One of the most significant inhibitors to effective interaction is the cognitive load it takes to memorize and perform precise, predefined inputs. Users are often forced to fit their movement into templates that have generalized any relatable physical nuance of their gesture. This prevents users from developing any kind of meaningful semiotic structure that gives them access to higher level relationships and more abstract concepts, often resorting to simple and direct connections between movement and ideas. In a fluid and dynamic interaction setting, especially where the development of a personalized language of movement would promote more established semiotic structures, this is not optimal. Users would be forced to learn a collective gesture library onto which they would have to project their expressions, remaining contained within impersonal constructs that are removed from their own conceptions. 2
To empower users to define their own gestural relationships with symbols, we have
implemented a novel system that nurtures an effective learning environment through
real-time analytics, fostering embodied cognition. We train a hybrid hierarchical
hidden-Markov model (HHMM) [
In order to collect motion data, we use the Microsoft Kinect V2 [
Our feature generation and data processing software is written in Max 6/Jitter [
From these features, we train a HHMM to recognize and extract abstract model
features from user gestures. The user is able to define as many gestures as they wish,
training each class in succession, using only one example for each class. Once the user
has recorded their complete gesture space, an HHMM adapted from the IRCAM
`MUBU' package [
After training, we combine these data streams create a hybrid model that classifies gestures, extracts abstract model parameters, defines kinematic features, and measures summary statistics. This model can then be used to recognize and analyze users' gestures, allowing them to begin developing semiotic structure between abstract concepts and their movement. While these streams all ultimately rely on a singular source, they provide unique analytic opportunities and are thus considered different modes. 3
While we have shown that our model can be successful in experimental tasks,
immersive simulation theatres, and several other applications [
The biggest challenge we face is determining the most relevant features that engage the user and provide analytic insight. While we can gather and analyze a multitude of features, determining the most prescient features and how those features interact is an opportunity for realizing a more effective model in performance and accuracy. For our initial work, we chose to create a robust model that incorporates as many features as we could collect in order to empirically determine the best feature set. We've experimented with variety of approaches to focusing our features including setting empirically determined thresholds, selecting only statistically significant features, and providing users with the agency to select features of their choosing. While we've found some success with these methods, more investigation needs to be done.
While we are able to extract a variety of features, connecting them to the users beyond intuitive responses has proven elusive. The ability to adapt our analytics in realtime relies heavily on an efficient mapping schema that takes full advantage of all of the features we are extracting, physical and abstract. Additionally, this is essential to creating a more intuitive and empowering interaction for the user. In order to address this, we've incorporated real-time, visual feedback into our training and simulations, reinforcing user interaction through performance. 4
The ability to ground abstract concepts in tangible, intuitive, embodied metaphors is a vital foray into advancing effective education and other application interventions. By creating a model in which the user can specifically and quickly define their interactions through the fusion of different data streams, we offer a novel tool that begins to explore one-shot learning as a means to real-time multimodal learning analytics.