HoloLearn: Using holograms to support naturalistic interaction in virtual classrooms ?

Abstract

Traditional online communications tools used in education are limited in terms of fostering naturalistic or life-like interaction. Such limited interactions in classrooms can negatively impact learning. Holograms are promising tools that show potential to overcome such limitations by a ording more life-like interactions in virtual classrooms. In this paper, we introduce the prototype built within the context of the project, "HoloLearn", which is currently ongoing and aims to foster lifelike interactions between teachers and students. Furthermore, we discuss the limitations of the current prototype and also the steps that need to be undertaken in the future.

Keywords

• Mixed reality
• Holograms
• Online Classes
• Immersive Technologies

Introduction

According to the survey conducted by the student council at TU Delft and the investigations of the 4TU Centre for Engineering Education, the Education & Student A airs of Wageningen University and Research, and the Education and Learning Sciences chair group, 68.1% of the students indicated that online education has a (very) negative e ect on their performance [ 1 ]. The study shows that the limited social interactions and engagements in online classrooms, lead to lack of energy and motivation in students. Correspondingly, educators who took part in the study also indicated that the lack of student interaction, and thereof engagement [ 3 ] and feedback, in online video conferences a ected their teaching.

Engagement in online classrooms is inherently more di cult than in colocated scenarios [ 2 ] which results in lower engagement. Furthermore, online classrooms via conventional media such as Skype™, Zoom™or YouSeeU™o er limited interactions between students themselves and between student and teacher. Limited interaction leads to lack of presence [ 3 ], a form of engagement, which in turn can negatively a ect learning [ 5 ] [ 4 ]. Michele [ 6 ] also accentuates several drawbacks of online classrooms such as the inability of the student to focus and interact with teachers and other students, which can also negatively a ect learning. Holograms as an immersive medium for online classrooms can potentially address these shortcomings [ 11 ] [ 12 ]. Holograms are 3D projections of volumetric objects such that it retains the objects properties such as depth, parallax etc, allowing it to blend in with the physical environment and provide a more realistic learning experience. In this paper, we introduce a prototype of an online classroom which utilises holographic technologies, built in the context of the project "HoloLearn" (Holographic Learning). It aims to improve presence in online classrooms with the help of holograms. We discuss the current status of the prototypical development and necessary future works in the following sections. 1.1

Background

The "Hololearn" project aims to develop the infrastructure required to conduct holographic online lectures at TU Delft. In addition to simply providing a new platform for conducting lectures, the project also aims to amplify presence in online classrooms by using holograms in education. Presence as a state of alert awareness, receptivity and connectedness to the mental, emotional and physical workings of both the individual and the group in the context of their learning environments and the ability to respond with a considered and compassionate best next step [ 7 ] [ 13 ]. Garrison et al. [ 9 ] identi ed three distinct elements of an educational experience | cognitive presence, social presence, and teaching presence. Li and Lefervre [ 12 ] argue that holograms can enhance teaching presence (broadly characterised as the virtual \visibility" of an instructor in an online learning environment [ 8 ]) and social presence (refers to their ability to present themselves and their characteristics to others) among students in online classrooms. Both teaching presence and social presence are vital for learning [ 14 ] [ 11 ]. Furthermore, with holograms, students can perceive the entirety of the teacher as being a part of their physical surrounding which helps in promoting teaching presence. Holograms also enable non-verbal signals such as posture and gestures in an online settings, which facilitates better communication [ 10 ], interaction and thus, engagement which leads to enhanced cognitive presence (related to learners' ability to construct meaning through communication). Therefore, it can be argued that amplifying presence in online classes can potentially lead to enhanced learning.

While concrete ndings on learning bene ts of using holograms in online lectures are lacking, multiple studies have shown bene ts in other attributes that can lead to better learning outcomes. Paredes et al. [ 14 ] reported that the students exposed to holographic teaching, experienced higher levels of learning ow experience, a state of full immersion which is an indicator of learning achievement [ 15 ], in comparison to traditional classes. Similarly, Li and Lefevre [ 12 ] also reported that engagement increased in the holographic seminar compared to non-holographic video conferencing. Engagement or presence, which refers to students' e ort and involvement in the learning activities, is also regarded as an important indicator of student performance [ 16 ]. As such, use of holograms in education shows potential for improving presence in online classrooms and therefore, also the learning outcomes. In the following, we describe the prototypical developments undertaken in the project "HoloLearn" in order to reap the bene ts of holographic lecture. 2

Holographic lecture

The services provided by the "HoloLearn" prototype must be be accessible to all students and support lectures with large number of students. Therefore, the project uses a browser based 3D classroom environment to allow students with varying computer resources to access lectures (see gure 1). However, it should be noted that the holographic lecture can be hosted in other ways such as within a virtual reality environment. However, due to virtual reality glasses currently not readily available for all the students, we decided to rst implement the browser based platform. The browser based classroom environment facilitates student to student interaction, displaying of the teacher's 3D model and sharing of on screen content along with audio channels. The goal in constructing the prototype classroom was to identify and assess di erent technologies that are capable of supporting such an experience. The prototype is comprised of separate facilities for data capture, data transmission and the construction of the classroom environment (see gure 2). The primary motivation for separating these tasks is to encourage modularity. In this way, it is possible to change the software or hardware resources used in one facility without making substantial changes in any other. The data capture component is primarily responsible for interfacing with speci c devices, performing necessary preprocessing and packaging the content before transmission. The transmission component ensures the correct routing of content streams between users. For example, the video content of the teacher is forwarded to all of the students' browsers and the teacher receives the video content of each student. Finally, the classroom, or perhaps better named "playback" component, is where all of the content culminates to create the 3D learning experience. All processing related to rendering and graphical assets takes place in the classroom component. The current facilities of the application are built for functionality and revolve around the speci cs of the interactive classroom use case scenario. Alterations in the functional or aesthetic design of any components to better suit a di erent scenario are of course possible.

Visual and LiDAR

Capture Device Specific

Interfacing Video Preprocessing

Data Stream Packaging Capture

Node.js + WebRTC

Broadcaster Session Manager RTC Connector

Transmission WebRTC Peer-Peer

Connections

Video Stream Reception

Dynamic 3D Model

Generation

Back-end 3D Environment

Rendering User Interface

Front-end Playback The prototype currently supports the use of any device from the Intel RealSense D and L product lines and the Microsoft Azure Kinect. All rendering of holograms and graphical computation is handled through Three.js [ 17 ], a Javascript library built upon WebGL.

Turning depth data into a volumetric model can be done in more than one way. Two techniques are used in this project: indexed mesh construction and oating point cloud. Each technique o ers di erent advantages and disadvantages. For instance, the mesh construction technique produces a model without any visual gaps but at the cost of increased computation for every frame. Alternatively, the oating point cloud is comparatively computationally simple but less visually complete (see gure 3). Users have the option to alter the appearance of the 3D model, for both methods of rendering, as seen by them in real time. This includes: toggling the visibility of the 3D model, changing between the two model types, changing the resolution of the 3D model and changing the position of the 3D model within the classroom. Point cloud capture, processing, transmission and modelling are all computationally expensive operations. Subsequently, it is bene cial to have a suitable alternative available for when the circumstances limit the usability and experience of the 3D model. For this reason, an alternate path in the form of joint and facial tracking is currently being explored. This approach is comparatively light weight and reduces network load as a much lesser volume of data is being transmitted. A similar result to that of a model constructed from a point cloud is achieved with the use of a rigged model (containing a skeletal structure) of the teacher and a dedicated facial geometry to better convey movements of the facial muscles and expressions (see Figure 4.)

Classroom and student experience

Students are able to see and hear one another within the virtual classroom environment facilitated by their browser. The placement of students is such that it mimics the rows of seating in a physical classroom. A chat facility is also available as an alternative to interacting with audio. Students are provided with the typical options of an online meeting platform, such as disabling their video or microphone. Most importantly, the students can view and listen to their teacher as a hologram within the virtual environment (see Figure 5). This allows more naturalistic interactions and communication between teachers and students.

Limitations and Future works

The prototypical developments in "HoloLearn" are still in progress. A number of improvements are in order. For example, LiDAR cameras are sensitive to interference from other sources of infrared light, the most prevalent of which being sun light. This prototype already employs image processing techniques to mitigate this e ect but it is far from perfect. Finding e cient ways to completely remove this interference in software would greatly increase the environment tolerance and thus improve the versatility and uses cases of the product. Furthermore, exploring methods that could increase the de nition of the 3D model while not substantially increasing the computation load is a key area of interest in furthering this project. Expanding the application to make use of multiple cameras to capture a better representation of the teacher is also an area of interest. Further improvements are also needed in transmission format. Data is currently exchanged in the form of video streams. Subsequently, the data is subject to the limitations of image encoding, speci cally compression, which tends to "muddy" the depth data. Solving this issue is onerous as moving over to lossless video transmission would con ict with the strict low latency requirement of this application (the WebRTC protocol [ 18 ] is currently used for this reason). A student side, software based mitigation technique is currently the most suitable solution. The joint and facial tracking use case eliminates this problem simply by nature of what data is being captured (visual data vs. hierarchical joint data).

References

1. Stevens , T. ( 2020 ). Update on research on Education in Covid-19 times. WUR . https://www.wur.nl/en/newsarticle/Update-on -research-on-Education-in-Covid19-times .htm.

2. Morrison-Smith , S. , & Ruiz , J. ( 2020 ). Challenges and barriers in virtual teams: a literature review . In SN Applied Sciences (Vol. 2, Issue 6) . Springer International Publishing. https://doi.org/10.1007/s42452-020-2801-5

3. Tu , C.-H. , & McIsaac , M. ( 2002 ). The Relationship of Social Presence and Interaction in Online Classes . American Journal of Distance Education , 16 ( 3 ), 131 { 150 . https://doi.org/10.1207/S15389286AJDE1603 2

4. Kyei-Blankson , L. , Ntuli , E. , & Donnelly , H. ( 2016 ). Establishing the Importance of Interaction and Presence to Student Learning in Online Environments . World Journal of Educational Research , 3 ( 1 ), 48. https://doi.org/10.22158/wjer.v3n1p48

5. Okita , S. Y. ( 2012 ). Social Interactions and Learning . In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning (pp. 3104 { 3107 ). Springer US. https://doi.org/10.1007/978-1- 4419 -1428-6 1770

6. De Michele , R. ( 2020 ). Bene ts, drawbacks and psychological impact of online lectures during quarantine due to Covid-19 pandemic . ACM International Conference Proceeding Series, 257 { 260 . https://doi.org/10.1145/3411170.3411265

7. Rodgers , C. R. , & Raider-Roth , M. B. ( 2006 ). Presence in teaching . Teachers and Teaching: theory and practice , 12 ( 3 ), 265 - 287 .

8. Garrison , D. R. ( 2021 ). Teaching Presence. E-Learning in the 21st Century , 87 { 100 . https://doi.org/10.4324/ 9781315667263 - 16

9. Garrison , D. R. , Anderson , T. , & Archer , W. ( 1999 ). Critical inquiry in a text-based environment: Computer conferencing in higher education . The Internet and Higher Education , 2 ( 2 {3), 87 { 105 .

10. McNeill , D. , ( 1992 ), Hand and Mind: What Gestures Reveal about Thought , Chicago: University of Chicago Press.

11. Luevano , E. , Lara , E. L. de, & Castro , J. E. ( 2015 ). Use of Telepresence and Holographic Projection Mobile Device for College Degree Level . Procedia Computer Science , 75 , 339{ 347 . https://doi.org/https://doi.org/10.1016/j.procs. 2015 . 12 .256

12. Li , N. , & Lefevre , D. ( 2020 ). Holographic teaching presence: participant experiences of interactive synchronous seminars delivered via holographic videoconferencing . Research in Learning Technology, 28 . https://doi.org/10.25304/rlt.v28. 2265

13. Carol R. Rodgers & Miriam B. Raider-Roth ( 2006 ) Presence in teaching , Teachers and Teaching , 12 : 3 , 265 - 287 , DOI: 10.1080/13450600500467548

14. Paredes , S. G. , & Vazquez , N. R. ( 2020 ). Is holographic teaching an educational innovation ? International Journal on Interactive Design and Manufacturing (IJIDeM) , 14 ( 4 ), 1321 { 1336 .

15. Joo , Y. J. , Lim , K. Y. , & Kim , S. M. ( 2012 ). A model for predicting learning ow and achievement in corporate e-learning . Educational Technology & Society , 15 ( 1 ), 313 .

16. Kuh , G. D. ( 2009 ). The national survey of student engagement: Conceptual and empirical foundations . New Directions for Institutional Research , 2009 ( 141 ), 5 { 20 .

17. Three . js { JavaScript 3D Library . Threejs.org. ( 2021 ). Retrieved 9 July 2021 , from https://threejs.org/.

18. WebRTC. WebRTC. ( 2021 ). Retrieved 9 July 2021 , from https://webrtc.org/.