HoloLearn: Using holograms to support naturalistic interaction in virtual classrooms ? Tristan Quin1[0000−0001−6113−2654] , Bibeg Limbu2[0000−0002−1269−6864] , Michel Beerens3,2 , and Marcus Specht1,2[0000−0002−6086−8480] 1 TU Delft, Netherlands 2 Leiden Delft Erasmus Center for Education and Learning, TU Delft, Netherlands 3 The NewMedia Center, TU Delft, Netherlands Abstract. Traditional online communications tools used in education are limited in terms of fostering naturalistic or life-like interaction. Such limited interactions in classrooms can negatively impact learning. Holo- grams are promising tools that show potential to overcome such limi- tations by affording more life-like interactions in virtual classrooms. In this paper, we introduce the prototype built within the context of the project, ”HoloLearn”, which is currently ongoing and aims to foster life- like interactions between teachers and students. Furthermore, we discuss the limitations of the current prototype and also the steps that need to be undertaken in the future. Keywords: Mixed reality · Holograms · Online Classes · Immersive Technologies 1 Introduction According to the survey conducted by the student council at TU Delft and the investigations of the 4TU Centre for Engineering Education, the Education & Student Affairs of Wageningen University and Research, and the Education and Learning Sciences chair group, 68.1% of the students indicated that online education has a (very) negative effect on their performance [1]. The study shows that the limited social interactions and engagements in online classrooms, lead to lack of energy and motivation in students. Correspondingly, educators who took part in the study also indicated that the lack of student interaction, and thereof engagement [3] and feedback, in online video conferences affected their teaching. Engagement in online classrooms is inherently more difficult than in co- located scenarios [2] which results in lower engagement. Furthermore, online classrooms via conventional media such as Skype™, Zoom™or YouSeeU™offer lim- ited interactions between students themselves and between student and teacher. Limited interaction leads to lack of presence [3], a form of engagement, which in turn can negatively affect learning [5] [4]. Michele [6] also accentuates several ? Funded by Student council of TU Delft Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 2 Quin et al. drawbacks of online classrooms such as the inability of the student to focus and interact with teachers and other students, which can also negatively affect learn- ing. Holograms as an immersive medium for online classrooms can potentially address these shortcomings [11] [12]. Holograms are 3D projections of volumet- ric objects such that it retains the objects properties such as depth, parallax etc, allowing it to blend in with the physical environment and provide a more realistic learning experience. In this paper, we introduce a prototype of an on- line classroom which utilises holographic technologies, built in the context of the project ”HoloLearn” (Holographic Learning). It aims to improve presence in online classrooms with the help of holograms. We discuss the current sta- tus of the prototypical development and necessary future works in the following sections. 1.1 Background The ”Hololearn” project aims to develop the infrastructure required to conduct holographic online lectures at TU Delft. In addition to simply providing a new platform for conducting lectures, the project also aims to amplify presence in online classrooms by using holograms in education. Presence as a state of alert awareness, receptivity and connectedness to the mental, emotional and physical workings of both the individual and the group in the context of their learning environments and the ability to respond with a considered and compassionate best next step [7] [13]. Garrison et al. [9] identified three distinct elements of an educational experience — cognitive presence, social presence, and teaching pres- ence. Li and Lefervre [12] argue that holograms can enhance teaching presence (broadly characterised as the virtual “visibility” of an instructor in an online learning environment [8]) and social presence (refers to their ability to present themselves and their characteristics to others) among students in online class- rooms. Both teaching presence and social presence are vital for learning [14] [11]. Furthermore, with holograms, students can perceive the entirety of the teacher as being a part of their physical surrounding which helps in promoting teaching presence. Holograms also enable non-verbal signals such as posture and gestures in an online settings, which facilitates better communication [10], interaction and thus, engagement which leads to enhanced cognitive presence (related to learners’ ability to construct meaning through communication). Therefore, it can be argued that amplifying presence in online classes can potentially lead to enhanced learning. While concrete findings on learning benefits of using holograms in online lec- tures are lacking, multiple studies have shown benefits in other attributes that can lead to better learning outcomes. Paredes et al. [14] reported that the stu- dents exposed to holographic teaching, experienced higher levels of learning flow experience, a state of full immersion which is an indicator of learning achieve- ment [15], in comparison to traditional classes. Similarly, Li and Lefevre [12] also reported that engagement increased in the holographic seminar compared to non-holographic video conferencing. Engagement or presence, which refers to students’ effort and involvement in the learning activities, is also regarded as HoloLearn 3 an important indicator of student performance [16]. As such, use of holograms in education shows potential for improving presence in online classrooms and therefore, also the learning outcomes. In the following, we describe the proto- typical developments undertaken in the project ”HoloLearn” in order to reap the benefits of holographic lecture. 2 Holographic lecture The services provided by the ”HoloLearn” prototype must be be accessible to all students and support lectures with large number of students. Therefore, the project uses a browser based 3D classroom environment to allow students with varying computer resources to access lectures (see figure 1). However, it should be noted that the holographic lecture can be hosted in other ways such as within a virtual reality environment. However, due to virtual reality glasses currently not readily available for all the students, we decided to first implement the browser based platform. The browser based classroom environment facilitates student to student interaction, displaying of the teacher’s 3D model and sharing of on screen content along with audio channels. The goal in constructing the prototype classroom was to identify and assess different technologies that are capable of supporting such an experience. Fig. 1. Teacher’s view of the classroom. 2.1 System architecture The prototype is comprised of separate facilities for data capture, data trans- mission and the construction of the classroom environment (see figure 2). The primary motivation for separating these tasks is to encourage modularity. In this way, it is possible to change the software or hardware resources used in 4 Quin et al. one facility without making substantial changes in any other. The data capture component is primarily responsible for interfacing with specific devices, perform- ing necessary preprocessing and packaging the content before transmission. The transmission component ensures the correct routing of content streams between users. For example, the video content of the teacher is forwarded to all of the students’ browsers and the teacher receives the video content of each student. Finally, the classroom, or perhaps better named ”playback” component, is where all of the content culminates to create the 3D learning experience. All processing related to rendering and graphical assets takes place in the classroom component. The current facilities of the application are built for functionality and revolve around the specifics of the interactive classroom use case scenario. Alterations in the functional or aesthetic design of any components to better suit a different scenario are of course possible. Visual and LiDAR Node.js + WebRTC Video Stream Reception Capture Broadcaster Dynamic 3D Model Generation Device Specific Back-end Session Manager Interfacing 3D Environment Video Preprocessing RTC Connector Rendering Transmission User Interface Data Stream Front-end Packaging WebRTC Peer-Peer Capture Connections Playback Fig. 2. High level architecture of the prototype application with expanded macro com- ponents. 2.2 3D modelling The prototype currently supports the use of any device from the Intel RealSense D and L product lines and the Microsoft Azure Kinect. All rendering of holo- grams and graphical computation is handled through Three.js [17], a Javascript library built upon WebGL. Turning depth data into a volumetric model can be done in more than one way. Two techniques are used in this project: indexed mesh construction and floating point cloud. Each technique offers different advantages and disadvan- tages. For instance, the mesh construction technique produces a model without any visual gaps but at the cost of increased computation for every frame. Alter- natively, the floating point cloud is comparatively computationally simple but HoloLearn 5 less visually complete (see figure 3). Users have the option to alter the appear- ance of the 3D model, for both methods of rendering, as seen by them in real time. This includes: toggling the visibility of the 3D model, changing between the two model types, changing the resolution of the 3D model and changing the position of the 3D model within the classroom. Fig. 3. Left: indexed mesh model, right: point cloud model 2.3 An Alternative Approach: Joint and Facial Tracking Point cloud capture, processing, transmission and modelling are all computa- tionally expensive operations. Subsequently, it is beneficial to have a suitable alternative available for when the circumstances limit the usability and experi- ence of the 3D model. For this reason, an alternate path in the form of joint and facial tracking is currently being explored. This approach is comparatively light weight and reduces network load as a much lesser volume of data is being transmitted. A similar result to that of a model constructed from a point cloud is achieved with the use of a rigged model (containing a skeletal structure) of the teacher and a dedicated facial geometry to better convey movements of the facial muscles and expressions (see Figure 4.) 6 Quin et al. Fig. 4. Rigged human model and facial mesh 2.4 Classroom and student experience Students are able to see and hear one another within the virtual classroom environment facilitated by their browser. The placement of students is such that it mimics the rows of seating in a physical classroom. A chat facility is also available as an alternative to interacting with audio. Students are provided with the typical options of an online meeting platform, such as disabling their video or microphone. Most importantly, the students can view and listen to their teacher as a hologram within the virtual environment (see Figure 5). This allows more naturalistic interactions and communication between teachers and students. Fig. 5. Student’s view of the holographic teacher during lecture. HoloLearn 7 3 Limitations and Future works The prototypical developments in ”HoloLearn” are still in progress. A number of improvements are in order. For example, LiDAR cameras are sensitive to inter- ference from other sources of infrared light, the most prevalent of which being sun light. This prototype already employs image processing techniques to mitigate this effect but it is far from perfect. Finding efficient ways to completely remove this interference in software would greatly increase the environment tolerance and thus improve the versatility and uses cases of the product. Furthermore, exploring methods that could increase the definition of the 3D model while not substantially increasing the computation load is a key area of interest in further- ing this project. Expanding the application to make use of multiple cameras to capture a better representation of the teacher is also an area of interest. Fur- ther improvements are also needed in transmission format. Data is currently exchanged in the form of video streams. Subsequently, the data is subject to the limitations of image encoding, specifically compression, which tends to ”muddy” the depth data. Solving this issue is onerous as moving over to lossless video transmission would conflict with the strict low latency requirement of this appli- cation (the WebRTC protocol [18] is currently used for this reason). A student side, software based mitigation technique is currently the most suitable solution. The joint and facial tracking use case eliminates this problem simply by nature of what data is being captured (visual data vs. hierarchical joint data). References 1. Stevens, T. (2020). Update on research on Education in Covid-19 times. WUR. https://www.wur.nl/en/newsarticle/Update-on-research-on-Education-in-Covid- 19-times.htm. 2. Morrison-Smith, S., & Ruiz, J. (2020). Challenges and barriers in virtual teams: a literature review. In SN Applied Sciences (Vol. 2, Issue 6). Springer International Publishing. https://doi.org/10.1007/s42452-020-2801-5 3. Tu, C.-H., & McIsaac, M. (2002). The Relationship of Social Presence and Inter- action in Online Classes. American Journal of Distance Education, 16(3), 131–150. https://doi.org/10.1207/S15389286AJDE1603 2 4. Kyei-Blankson, L., Ntuli, E., & Donnelly, H. (2016). Establishing the Importance of Interaction and Presence to Student Learning in Online Environments. World Journal of Educational Research, 3(1), 48. https://doi.org/10.22158/wjer.v3n1p48 5. Okita, S. Y. (2012). Social Interactions and Learning. In N. M. Seel (Ed.), Encyclopedia of the Sciences of Learning (pp. 3104–3107). Springer US. https://doi.org/10.1007/978-1-4419-1428-6 1770 6. De Michele, R. (2020). Benefits, drawbacks and psychological impact of online lec- tures during quarantine due to Covid-19 pandemic. ACM International Conference Proceeding Series, 257–260. https://doi.org/10.1145/3411170.3411265 7. Rodgers, C. R., & Raider-Roth, M. B. (2006). Presence in teaching. Teachers and Teaching: theory and practice, 12(3), 265-287. 8. Garrison, D. R. (2021). Teaching Presence. E-Learning in the 21st Century, 87–100. https://doi.org/10.4324/9781315667263-16 8 Quin et al. 9. Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2–3), 87–105. 10. McNeill, D., (1992), Hand and Mind: What Gestures Reveal about Thought, Chicago: University of Chicago Press. 11. Luévano, E., Lara, E. L. de, & Castro, J. E. (2015). Use of Telepresence and Holographic Projection Mobile Device for College Degree Level. Procedia Computer Science, 75, 339–347. https://doi.org/https://doi.org/10.1016/j.procs.2015.12.256 12. Li, N., & Lefevre, D. (2020). Holographic teaching presence: participant experiences of interactive synchronous seminars delivered via holographic videoconferencing. Research in Learning Technology, 28. https://doi.org/10.25304/rlt.v28.2265 13. Carol R. Rodgers & Miriam B. Raider-Roth (2006) Presence in teaching, Teachers and Teaching, 12:3, 265-287, DOI: 10.1080/13450600500467548 14. Paredes, S. G., & Vázquez, N. R. (2020). Is holographic teaching an educational in- novation? International Journal on Interactive Design and Manufacturing (IJIDeM), 14(4), 1321–1336. 15. Joo, Y. J., Lim, K. Y., & Kim, S. M. (2012). A model for predicting learning flow and achievement in corporate e-learning. Educational Technology & Society, 15(1), 313. 16. Kuh, G. D. (2009). The national survey of student engagement: Conceptual and empirical foundations. New Directions for Institutional Research, 2009(141), 5–20. 17. Three.js – JavaScript 3D Library. Threejs.org. (2021). Retrieved 9 July 2021, from https://threejs.org/. 18. WebRTC. WebRTC. (2021). Retrieved 9 July 2021, from https://webrtc.org/.