=Paper=
{{Paper
|id=Vol-2979/paper4
|storemode=property
|title=HoloLearn: Using holograms to support naturalistic interaction in virtual classrooms
|pdfUrl=https://ceur-ws.org/Vol-2979/paper4.pdf
|volume=Vol-2979
|authors=Tristan Quin,Bibeg Limbu,Michel Beerens,Marcus Specht
|dblpUrl=https://dblp.org/rec/conf/ectel/QuinLBS21
}}
==HoloLearn: Using holograms to support naturalistic interaction in virtual classrooms==
https://ceur-ws.org/Vol-2979/paper4.pdf
HoloLearn: Using holograms to support
naturalistic interaction in virtual classrooms ?
Tristan Quin1[0000−0001−6113−2654] , Bibeg Limbu2[0000−0002−1269−6864] , Michel
Beerens3,2 , and Marcus Specht1,2[0000−0002−6086−8480]
1
TU Delft, Netherlands
2
Leiden Delft Erasmus Center for Education and Learning, TU Delft, Netherlands
3
The NewMedia Center, TU Delft, Netherlands
Abstract. Traditional online communications tools used in education
are limited in terms of fostering naturalistic or life-like interaction. Such
limited interactions in classrooms can negatively impact learning. Holo-
grams are promising tools that show potential to overcome such limi-
tations by affording more life-like interactions in virtual classrooms. In
this paper, we introduce the prototype built within the context of the
project, ”HoloLearn”, which is currently ongoing and aims to foster life-
like interactions between teachers and students. Furthermore, we discuss
the limitations of the current prototype and also the steps that need to
be undertaken in the future.
Keywords: Mixed reality · Holograms · Online Classes · Immersive
Technologies
1 Introduction
According to the survey conducted by the student council at TU Delft and
the investigations of the 4TU Centre for Engineering Education, the Education
& Student Affairs of Wageningen University and Research, and the Education
and Learning Sciences chair group, 68.1% of the students indicated that online
education has a (very) negative effect on their performance [1]. The study shows
that the limited social interactions and engagements in online classrooms, lead
to lack of energy and motivation in students. Correspondingly, educators who
took part in the study also indicated that the lack of student interaction, and
thereof engagement [3] and feedback, in online video conferences affected their
teaching.
Engagement in online classrooms is inherently more difficult than in co-
located scenarios [2] which results in lower engagement. Furthermore, online
classrooms via conventional media such as Skype™, Zoom™or YouSeeU™offer lim-
ited interactions between students themselves and between student and teacher.
Limited interaction leads to lack of presence [3], a form of engagement, which
in turn can negatively affect learning [5] [4]. Michele [6] also accentuates several
?
Funded by Student council of TU Delft
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
2 Quin et al.
drawbacks of online classrooms such as the inability of the student to focus and
interact with teachers and other students, which can also negatively affect learn-
ing. Holograms as an immersive medium for online classrooms can potentially
address these shortcomings [11] [12]. Holograms are 3D projections of volumet-
ric objects such that it retains the objects properties such as depth, parallax
etc, allowing it to blend in with the physical environment and provide a more
realistic learning experience. In this paper, we introduce a prototype of an on-
line classroom which utilises holographic technologies, built in the context of
the project ”HoloLearn” (Holographic Learning). It aims to improve presence
in online classrooms with the help of holograms. We discuss the current sta-
tus of the prototypical development and necessary future works in the following
sections.
1.1 Background
The ”Hololearn” project aims to develop the infrastructure required to conduct
holographic online lectures at TU Delft. In addition to simply providing a new
platform for conducting lectures, the project also aims to amplify presence in
online classrooms by using holograms in education. Presence as a state of alert
awareness, receptivity and connectedness to the mental, emotional and physical
workings of both the individual and the group in the context of their learning
environments and the ability to respond with a considered and compassionate
best next step [7] [13]. Garrison et al. [9] identified three distinct elements of an
educational experience — cognitive presence, social presence, and teaching pres-
ence. Li and Lefervre [12] argue that holograms can enhance teaching presence
(broadly characterised as the virtual “visibility” of an instructor in an online
learning environment [8]) and social presence (refers to their ability to present
themselves and their characteristics to others) among students in online class-
rooms. Both teaching presence and social presence are vital for learning [14] [11].
Furthermore, with holograms, students can perceive the entirety of the teacher
as being a part of their physical surrounding which helps in promoting teaching
presence. Holograms also enable non-verbal signals such as posture and gestures
in an online settings, which facilitates better communication [10], interaction
and thus, engagement which leads to enhanced cognitive presence (related to
learners’ ability to construct meaning through communication). Therefore, it
can be argued that amplifying presence in online classes can potentially lead to
enhanced learning.
While concrete findings on learning benefits of using holograms in online lec-
tures are lacking, multiple studies have shown benefits in other attributes that
can lead to better learning outcomes. Paredes et al. [14] reported that the stu-
dents exposed to holographic teaching, experienced higher levels of learning flow
experience, a state of full immersion which is an indicator of learning achieve-
ment [15], in comparison to traditional classes. Similarly, Li and Lefevre [12]
also reported that engagement increased in the holographic seminar compared
to non-holographic video conferencing. Engagement or presence, which refers to
students’ effort and involvement in the learning activities, is also regarded as
HoloLearn 3
an important indicator of student performance [16]. As such, use of holograms
in education shows potential for improving presence in online classrooms and
therefore, also the learning outcomes. In the following, we describe the proto-
typical developments undertaken in the project ”HoloLearn” in order to reap
the benefits of holographic lecture.
2 Holographic lecture
The services provided by the ”HoloLearn” prototype must be be accessible to
all students and support lectures with large number of students. Therefore, the
project uses a browser based 3D classroom environment to allow students with
varying computer resources to access lectures (see figure 1). However, it should be
noted that the holographic lecture can be hosted in other ways such as within a
virtual reality environment. However, due to virtual reality glasses currently not
readily available for all the students, we decided to first implement the browser
based platform. The browser based classroom environment facilitates student
to student interaction, displaying of the teacher’s 3D model and sharing of on
screen content along with audio channels. The goal in constructing the prototype
classroom was to identify and assess different technologies that are capable of
supporting such an experience.
Fig. 1. Teacher’s view of the classroom.
2.1 System architecture
The prototype is comprised of separate facilities for data capture, data trans-
mission and the construction of the classroom environment (see figure 2). The
primary motivation for separating these tasks is to encourage modularity. In
this way, it is possible to change the software or hardware resources used in
4 Quin et al.
one facility without making substantial changes in any other. The data capture
component is primarily responsible for interfacing with specific devices, perform-
ing necessary preprocessing and packaging the content before transmission. The
transmission component ensures the correct routing of content streams between
users. For example, the video content of the teacher is forwarded to all of the
students’ browsers and the teacher receives the video content of each student.
Finally, the classroom, or perhaps better named ”playback” component, is where
all of the content culminates to create the 3D learning experience. All processing
related to rendering and graphical assets takes place in the classroom component.
The current facilities of the application are built for functionality and revolve
around the specifics of the interactive classroom use case scenario. Alterations
in the functional or aesthetic design of any components to better suit a different
scenario are of course possible.
Visual and LiDAR Node.js + WebRTC Video Stream Reception
Capture Broadcaster
Dynamic 3D Model
Generation
Device Specific Back-end
Session Manager
Interfacing
3D Environment
Video Preprocessing RTC Connector Rendering
Transmission User Interface
Data Stream
Front-end
Packaging
WebRTC Peer-Peer
Capture Connections Playback
Fig. 2. High level architecture of the prototype application with expanded macro com-
ponents.
2.2 3D modelling
The prototype currently supports the use of any device from the Intel RealSense
D and L product lines and the Microsoft Azure Kinect. All rendering of holo-
grams and graphical computation is handled through Three.js [17], a Javascript
library built upon WebGL.
Turning depth data into a volumetric model can be done in more than one
way. Two techniques are used in this project: indexed mesh construction and
floating point cloud. Each technique offers different advantages and disadvan-
tages. For instance, the mesh construction technique produces a model without
any visual gaps but at the cost of increased computation for every frame. Alter-
natively, the floating point cloud is comparatively computationally simple but
HoloLearn 5
less visually complete (see figure 3). Users have the option to alter the appear-
ance of the 3D model, for both methods of rendering, as seen by them in real
time. This includes: toggling the visibility of the 3D model, changing between
the two model types, changing the resolution of the 3D model and changing the
position of the 3D model within the classroom.
Fig. 3. Left: indexed mesh model, right: point cloud model
2.3 An Alternative Approach: Joint and Facial Tracking
Point cloud capture, processing, transmission and modelling are all computa-
tionally expensive operations. Subsequently, it is beneficial to have a suitable
alternative available for when the circumstances limit the usability and experi-
ence of the 3D model. For this reason, an alternate path in the form of joint
and facial tracking is currently being explored. This approach is comparatively
light weight and reduces network load as a much lesser volume of data is being
transmitted. A similar result to that of a model constructed from a point cloud
is achieved with the use of a rigged model (containing a skeletal structure) of
the teacher and a dedicated facial geometry to better convey movements of the
facial muscles and expressions (see Figure 4.)
6 Quin et al.
Fig. 4. Rigged human model and facial mesh
2.4 Classroom and student experience
Students are able to see and hear one another within the virtual classroom
environment facilitated by their browser. The placement of students is such that
it mimics the rows of seating in a physical classroom. A chat facility is also
available as an alternative to interacting with audio. Students are provided with
the typical options of an online meeting platform, such as disabling their video or
microphone. Most importantly, the students can view and listen to their teacher
as a hologram within the virtual environment (see Figure 5). This allows more
naturalistic interactions and communication between teachers and students.
Fig. 5. Student’s view of the holographic teacher during lecture.
HoloLearn 7
3 Limitations and Future works
The prototypical developments in ”HoloLearn” are still in progress. A number of
improvements are in order. For example, LiDAR cameras are sensitive to inter-
ference from other sources of infrared light, the most prevalent of which being sun
light. This prototype already employs image processing techniques to mitigate
this effect but it is far from perfect. Finding efficient ways to completely remove
this interference in software would greatly increase the environment tolerance
and thus improve the versatility and uses cases of the product. Furthermore,
exploring methods that could increase the definition of the 3D model while not
substantially increasing the computation load is a key area of interest in further-
ing this project. Expanding the application to make use of multiple cameras to
capture a better representation of the teacher is also an area of interest. Fur-
ther improvements are also needed in transmission format. Data is currently
exchanged in the form of video streams. Subsequently, the data is subject to the
limitations of image encoding, specifically compression, which tends to ”muddy”
the depth data. Solving this issue is onerous as moving over to lossless video
transmission would conflict with the strict low latency requirement of this appli-
cation (the WebRTC protocol [18] is currently used for this reason). A student
side, software based mitigation technique is currently the most suitable solution.
The joint and facial tracking use case eliminates this problem simply by nature
of what data is being captured (visual data vs. hierarchical joint data).
References
1. Stevens, T. (2020). Update on research on Education in Covid-19 times. WUR.
https://www.wur.nl/en/newsarticle/Update-on-research-on-Education-in-Covid-
19-times.htm.
2. Morrison-Smith, S., & Ruiz, J. (2020). Challenges and barriers in virtual teams: a
literature review. In SN Applied Sciences (Vol. 2, Issue 6). Springer International
Publishing. https://doi.org/10.1007/s42452-020-2801-5
3. Tu, C.-H., & McIsaac, M. (2002). The Relationship of Social Presence and Inter-
action in Online Classes. American Journal of Distance Education, 16(3), 131–150.
https://doi.org/10.1207/S15389286AJDE1603 2
4. Kyei-Blankson, L., Ntuli, E., & Donnelly, H. (2016). Establishing the Importance
of Interaction and Presence to Student Learning in Online Environments. World
Journal of Educational Research, 3(1), 48. https://doi.org/10.22158/wjer.v3n1p48
5. Okita, S. Y. (2012). Social Interactions and Learning. In N. M. Seel (Ed.),
Encyclopedia of the Sciences of Learning (pp. 3104–3107). Springer US.
https://doi.org/10.1007/978-1-4419-1428-6 1770
6. De Michele, R. (2020). Benefits, drawbacks and psychological impact of online lec-
tures during quarantine due to Covid-19 pandemic. ACM International Conference
Proceeding Series, 257–260. https://doi.org/10.1145/3411170.3411265
7. Rodgers, C. R., & Raider-Roth, M. B. (2006). Presence in teaching. Teachers and
Teaching: theory and practice, 12(3), 265-287.
8. Garrison, D. R. (2021). Teaching Presence. E-Learning in the 21st Century, 87–100.
https://doi.org/10.4324/9781315667263-16
8 Quin et al.
9. Garrison, D. R., Anderson, T., & Archer, W. (1999). Critical inquiry in a text-based
environment: Computer conferencing in higher education. The Internet and Higher
Education, 2(2–3), 87–105.
10. McNeill, D., (1992), Hand and Mind: What Gestures Reveal about Thought,
Chicago: University of Chicago Press.
11. Luévano, E., Lara, E. L. de, & Castro, J. E. (2015). Use of Telepresence and
Holographic Projection Mobile Device for College Degree Level. Procedia Computer
Science, 75, 339–347. https://doi.org/https://doi.org/10.1016/j.procs.2015.12.256
12. Li, N., & Lefevre, D. (2020). Holographic teaching presence: participant experiences
of interactive synchronous seminars delivered via holographic videoconferencing.
Research in Learning Technology, 28. https://doi.org/10.25304/rlt.v28.2265
13. Carol R. Rodgers & Miriam B. Raider-Roth (2006) Presence in teaching, Teachers
and Teaching, 12:3, 265-287, DOI: 10.1080/13450600500467548
14. Paredes, S. G., & Vázquez, N. R. (2020). Is holographic teaching an educational in-
novation? International Journal on Interactive Design and Manufacturing (IJIDeM),
14(4), 1321–1336.
15. Joo, Y. J., Lim, K. Y., & Kim, S. M. (2012). A model for predicting learning flow
and achievement in corporate e-learning. Educational Technology & Society, 15(1),
313.
16. Kuh, G. D. (2009). The national survey of student engagement: Conceptual and
empirical foundations. New Directions for Institutional Research, 2009(141), 5–20.
17. Three.js – JavaScript 3D Library. Threejs.org. (2021). Retrieved 9 July 2021, from
https://threejs.org/.
18. WebRTC. WebRTC. (2021). Retrieved 9 July 2021, from https://webrtc.org/.