Towards a mixed-reality tool for collaborative mind-maps

Abstract

Traditional user interfaces such as WIMP (Windows Icon Menu Pointer) are widespread in schools. They have been proven to be useful in many scenarios, yet they are unfit in the case of collaborative learning. This work aims to understand, design and prototype reality-based interactions to enable collaborative learning. To do so, we first have to understand the key components, and particularly, the cognitive processes underlying such interfaces and how users and artefacts interact with each other. Then, we will implement a tangible and augmented interface supporting learning in school context based on conceptual frameworks and literature. This talk will present our ongoing work on this direction.

-

Beyond WIMP interfaces, reality-based interaction (RbI) “increases the realism of interface objects and allow users to interact even more directly with them using actions that correspond to daily practices within the non-digital world.” [ 8 ]. It also provides hands-on activity and gestures which enhance the user’s learning abilities [6]. Good examples of RbI are Tangible User Interfaces (TUI, physical handles Copyright is held by the author/owner(s). manipulating virtual information) [ 13 ] and spatial augmented reality (SAR, complementing the real world with digital content) [ 11 ]. Both SAR and TUI are known to support collaborative activities like learning and problem solving [4].

Our goal is to go towards learning interfaces that support collaborative learning. In this work, we are interested in exploring new forms of interactions such as reality-based interaction. To this end we will need first to understand, from a cognitive science point of view, what are the determinants enabling interfaces to propose effective collaborative learning activities. Then, we will design interactions with the guidelines from previous work and frameworks in cognitive science. Finally, we will implement our interactions and propose new reality-based interfaces.

To foster collaborative learning, we chose to implement the mind-map technique [1], which allows users spatializing, categorising and sorting items, to create a visually organised graph of information. Mind-maps have a variety of usages, such as studying, planning, critical thinking and problem solving [2].

This work is part of a broader project called e-Tac funded by the French education and research ministry. E-tac aims at exploring how tangible and augmented interactions can benefit to collaborative learning in school contexts from elementary school to middle-school.

UNDERSTANDING So far, the literature has not fully identified the cognitive processes underlying tangible and augmented interactions [ 13 ], such as learning (alone or in collaboration). Some studies [ 12, 14 ] stress the importance of multi-sensory perception and movements to enhance performances with RbI. To be able to design interactions that support learning, we first have to understand which cognitive processes are involved when manipulating reality-based interfaces. This research project aims to contribute to a better understanding of cognition in the field of HCI. Cognitive science could shed light on HCI researchers to explore which cognitive processes are solicited using tangible and augmented interfaces in a learning task while manipulating new contexts. To this end, we will explore frameworks like representation and division of space around a subject (i.e. peri-personal and extra-personal space) [3] and how these different spaces are used to specialise information on a tabletop during a learning task. Moreover, collaborative interactions will be used on our interface and we also have to consider this while designing the system. Cognitive science provides frameworks to study these types of complex relations like the distributed cognition framework [7],” which considers a collaborative activity as taking place across individuals, artifacts and internal or external representations, as one cognitive system” [ 16 ].

DESIGNING Using cognitive frameworks allows us to design a set of interactions built to fit together both at individual and collective scale. By using tabletop interactions we can create a common space for users and the interface. This horizontal surface will allow us to design interaction with traditional tools like pen and paper. Hands-on activities above the surface can also be supported. Such as Papart [ 9 ] and TinkerLamp, our system [5] provide projections into the surface, supporting our system manipulation of digital objects and media.

PROTOTYPING Once we will better understand the process involved, then we will be able to iteratively design and prototype relevant interactions. For this, the hardware components of our interactive system should enable object tracking as well as fiducial markers on a tabletop to support spacialisation of information. To do so, we will use a video-projector to display content above a surface and optical based tracking systems to track objects.

This talk will present our ongoing work towards the understanding of the cognitive science frameworks related to our project on collaborative learning with RbI, such as distributed cognition, embodied cognition and spatial representation. This talk will also allow us to present our interface currently being developed as well as the results of the pilot study on the interface’s learning capacities which will be conducted before this talk.

evaluating orchestration technologies for the classroom. In European Conference on Technology Enhanced Learning, Springer (2012), 65–78.

Hollan, J., Hutchins, E., and Kirsh, D. Distributed cognition: toward a new foundation for humancomputer interaction research. ACM Transactions on Computer-Human Interaction (TOCHI) 7, 2 (2000), 174–196.

References

Goldin-Meadow , S. How gesture promotes learning throughout childhood . Child development perspectives 3 , 2 ( 2009 ), 106 - 111 .

8. Jacob , R. J. , Girouard , A. , Hirshfield , L. M. , Horn , M. S. , Shaer , O. , Solovey , E. T. , and Zigelbaum , J. Reality-based interaction: a framework for post-wimp interfaces . In Proceedings of the SIGCHI conference on Human factors in computing systems , ACM ( 2008 ), 201 - 210 .

9. Laviole , J. , and Hachet , M. Spatial augmented reality for physical drawing . In Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology , ACM ( 2012 ), 9 - 10 .

10. Marshall , P. Do tangible interfaces enhance learning ? In Proceedings of the 1st international conference on Tangible and embedded interaction , ACM ( 2007 ), 163 - 170 .

11. Raskar , R. , and Low , K.-L. Interacting with spatially augmented reality . In Proceedings of the 1st international conference on Computer graphics, virtual reality and visualisation , ACM ( 2001 ), 101 - 108 .

12. Schneider , B. , Sharma , K. , Cuendet , S. , Zufferey , G. , Dillenbourg , P. , and Pea , R. Using mobile eye-trackers to unpack the perceptual benefits of a tangible user interface for collaborative learning . ACM Transactions on Computer-Human Interaction (TOCHI) 23 , 6 ( 2016 ), 39 .

13. Shaer , O. , and Hornecker , E. Tangible user interfaces: past, present, and future directions . Foundations and Trends in Human-Computer Interaction 3 , 1 - 2 ( 2010 ), 1 - 137 .

14. Skulmowski , A. , Pradel , S. , Ku ̈hnert, T., Brunnett , G. , and Rey , G. D. Embodied learning using a tangible user interface: The effects of haptic perception and selective pointing on a spatial learning task . Computers & Education 92 ( 2016 ), 64 - 75 .

15. Stanton , D. , and Neale , H. . The effects of multiple mice on children's talk and interaction . Journal of Computer Assisted Learning 19, 2 ( 2003 ), 229 - 238 .

16. Vasiliou , C. , Ioannou , A. , and Zaphiris , P. Understanding collaborative learning activities in an information ecology: A distributed cognition account . Computers in Human Behavior 41 ( 2014 ), 544 - 553 .