=Paper=
{{Paper
|id=Vol-3657/paper2
|storemode=property
|title=Progressive Education: Augmented Reality Visualization of Human Anatomy on Hololens
2
|pdfUrl=https://ceur-ws.org/Vol-3657/paper2.pdf
|volume=Vol-3657
|authors=Aleksandr Shishkov,Simon Kolmanič
|dblpUrl=https://dblp.org/rec/conf/hci-si/ShishkovK23
}}
==Progressive Education: Augmented Reality Visualization of Human Anatomy on Hololens
2==
https://ceur-ws.org/Vol-3657/paper2.pdf
Progressive Education: Augmented Reality
Visualization of Human Anatomy on HoloLens 2
Aleksandr Shishkov1 , Simon Kolmanič1
1
University of Maribor, Faculty of Electrical Engineering and Computer Science, Koroška cesta 46, 2000 Maribor,
Slovenija
Abstract
The landscape of medical education is undergoing a transformative shift driven by rapid technological
evolution. In the 21st century, innovative platforms, notably Augmented Reality (AR), play a crucial role in
creating immersive and interactive learning environments, especially in medicine. The HoloLens headset,
representing a leading AR platform, stands out in revolutionizing medical education by seamlessly
merging the physical and digital realms. This article explores the development of an AR application for
HoloLens 2, specifically focusing on visualizing human anatomy and highlighting intricate details of
implants within the human system. The primary objective is to enrich the learning experience, providing
a comprehensive understanding of anatomical structures through the lens of cutting-edge technology.
Keywords
Human anatomy, Augmented reality, Microsoft HoloLens 2
1. Introduction
The rapid evolution of technology has always been a driving force behind advancements in
medical education. The immersion and interactive capabilities offered by Augmented Reality
(AR) headsets like the HoloLens represent a revolutionary step in the direction of enhancing
learning and research experiences in the medical field.
One significant area where AR can make a difference is in the visualization and understanding
of human anatomy and physiology. Traditional methods, such as textbooks or even 3D models,
provide a limited perspective. On the other hand, AR allows users to delve deep into the human
body, inspecting organs, systems, and even implants in a comprehensive and interactive manner.
Furthermore, with the increasing use of implants in medical treatments and surgeries, there’s
a growing need for a detailed, immersive visualization tool that can aid in both educational
and practical scenarios. Training healthcare professionals about the location, function, and
intricacies of these implants becomes paramount.
1.1. Progressive education
Progressive education is an educational philosophy that prioritizes student-centered learning,
hands-on engagement, collaboration, and project-based learning. This approach is crucial for
equipping students with the skills necessary for success in the 21st century. By shifting the focus
HCI SI 2023: Human-Computer Interaction Slovenia 2023, January 26, 2024, Maribor, Slovenia
$ aleksandr.shishkov@student.um.si (A. Shishkov); simon.kolmanic@um.si (S. Kolmanič)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
from traditional, teacher-centered methods to more interactive and participatory approaches,
progressive education fosters the development of critical skills essential for navigating the
challenges of the modern world. [1]
The developed AR application aligns with several principles of progressive education, foster-
ing the development of 21st-century skills among users. Among these skills, we would like to
emphasize:
• Critical thinking
• Creativity and innovation
• Self-direction and independence
• Using technology as a learning tool
To attain this goal, Progressive Education incorporates the following principles:
• Curriculum tailored to individual learning styles, developmental needs, and intellectual
interests
• The student as an active partner in learning
• Arts, sciences, and humanities equally valued in an interdisciplinary curriculum
• Learning through direct experience and primary material
• A focus on multicultural and global perspectives
• The school as a model of democracy
The emphasis on critical thinking, creativity, and innovation is evident in the user-friendly
design that encourages seamless interaction and exploration of the anatomical model. The ap-
plication’s accessibility promotes self-direction and independence in learning, allowing learners
to take ownership of their educational journey. By leveraging AR technology as a learning tool,
the application transforms the educational experience into an engaging and interactive journey.
The principles of progressive education further guide the application’s development, ensuring
a tailored curriculum that accommodates different learning styles, encourages students to be
active partners in their learning, values interdisciplinary perspectives, provides direct hands-on
experiences, embraces multicultural perspectives, and reflects collaborative and democratic
principles.
2. Objectives
To achieve the above aim we developed an educational application to visualize human anatomy
for the HoloLens 2. The application’s centerpiece is a realistic human model encompassing
all body systems—skin, skeleton, muscles, organs, and many others. Our primary goal was to
design an easily comprehensible and user-friendly application, promoting seamless interaction
and exploration of the anatomical model.
During the design phase, we envisioned a layered representation of various body systems, al-
lowing users to toggle between them. As development progressed, additional concepts emerged,
such as organ search functionality and an enhanced preview of human organs.
Figure 1: Example of application start view.
At this moment, the application itself consists of a single expansive scene where the human
model is positioned one meter away from the user. On the foreground is the main menu,
incorporating all essential functions of the application, as depicted in Figure 1. It includes
four tabs on the left side of the canvas, which user can click and change the display of the
corresponding tab content:
• Body layer switching: This menu features a grid of toggling buttons with layer icon and
name, that shows corresponding layers.
• Search: Includes an input box, where user can enter desired query and a list of found
items. User can click on any item and open detailed object overview as illustrated in
Figure 2.
• Popular organs or bones: Displays a list of the most popular organs (e.g, heart, brain,
skull), that also opens detailed object overview.
• Settings: Includes utility buttons, for example, to move the model across the space and
toggle animations.
3. Challenges and solutions
To implement our ideas effectively, we needed to carefully select a comprehensive human model
that could meet our specific requirements. After exploring the market for available human
models, we chose a suitable option and proceeded with it. This model, intricately designed in
Blender, a powerful open-source 3D content creation tool, encompasses all human systems,
Figure 2: Detailed object overview.
including muscular, organ, lymphatic, skeletal, skin, and circulatory systems, see Figure 3.
Additionally, it features an armature and a sample animation, providing a detailed and well-
structured representation. This highly refined model presented an ideal foundation for our
project. Our initial and crucial task involved seamlessly importing this detailed model into
Unity, the chosen game engine for developing our HoloLens application.
Later in the development process, we expanded the model’s capabilities by incorporating
various implants, such as an electrode grid, infusion pump, neurostimulator, and electroen-
cephalogram, as shown on middle object in Figure 3. These additions further enriched the
application, allowing users to explore and understand the integration of these implants within
the human body’s intricate systems.
The general method for importing 3D models into Unity involves using the FBX extension.
However, despite being a connecting element, challenges often arise due to the absence of certain
functionalities during the import process, particularly in the development of our HoloLens
application. In response, we prepared different solutions to overcome these obstacles.
The intricate material system of Blender posed difficulties during the conversion to Unity.
The FBX extension faces compatibility issues with Blender’s complex materials. To overcome
this, our solution involved the strategic use of Python scripts within Blender. Python scripting in
Blender is a powerful tool that allows users to program specific functions to suit their needs. In
this instance, our scripts played a crucial role by baking textures, ensuring a smooth transition
of materials into Unity.
Another significant obstacle revolved around the loss of object hierarchy during the import
process. Blender organizes objects into collections, creating a hierarchical structure. However,
Unity does not inherently support this hierarchy, resulting in a disorganized structure. Once
Figure 3: Different model body systems (https://goo.gl/VLCRBB).
again, Python scripts proved invaluable. We developed a script that renamed objects based on
their collection, preserving a similar hierarchy in Unity.
Optimizing complex systems, such as the skeleton or organs with numerous objects and
polygons, presented a third challenge. Leveraging Blender’s Decimate modifier, which reduces
polygon counts, and a script that consolidated disconnected objects into a single structure
proved instrumental in overcoming the optimization challenge on the HoloLens platform.
As a result, we have several scripts, that had the most influence on the process:
1. Bake Collections Objects: Iterates over all the collections and objects inside and bake
diffuse and normal texture for every object separately.
2. Rename Collection Objects: Iterates over all the objects and appends the collection name
to the object.
3. Collection Joiner: Selects all the objects of collection and joins them to a single object.
These scripts reference the Blender Python API, which provides extensive resources for
developers.
With continuous refinement and testing of our scripts, we successfully imported the model
into Unity, paving the way for subsequent stages of development. The result was a realistic
human model seamlessly incorporating various body systems.
3.1. Unity Application Development and MRTK Library Features
For the development of augmented reality applications on the Unity platform, the powerful
Mixed Reality Toolkit (MRTK) comes into play. This library significantly simplifies the develop-
ment process by abstracting away low-level functions. It encompasses various UI components,
user interaction functions, physics simulation, and more. [2]
At the time of creating our application, MRTK had already reached its third global version
and was actively evolving. Our project was implemented using the second version of the library,
where we discovered and utilized numerous useful functions. MRTK’s documentation provides
an excellent resource with an extensive set of examples and educational materials, serving as
inspiration for developers.
In the scope of our application, we aimed to create a maximally intuitive user experience
and interface. The educational materials of the library helped us master the controls, enabling
us to work towards developing an optimal UI. An example of this approach is the main menu
where clear controls contribute to a user experience that is easy to understand. Main application
functions are easily discernible, and interaction with the primary model is straightforward.
In our project, we leveraged several MRTK functions and widgets to enhance the user
experience:
1. Bake Collections Objects: Iterates over all the collections and objects inside and bake
diffuse and normal texture for every object separately.
2. Rename Collection Objects: Iterates over all the objects and appends the collection name
to the object.
3. Collection Joiner: Selects all the objects of collection and joins them to a single object.
4. Bounds Control and Object Manipulator: These were utilized to manipulate the human
model, allowing users to interact with and explore anatomical structures seamlessly.
5. Directional Indicator and Solver Handler: These components were employed for UI/UX
enhancements, providing clear directional cues, and handling problem-solving aspects.
6. Interaction and Manipulation Scripts for UI Components: These scripts facilitated the
interaction and manipulation of various UI elements within the application.
7. Scrollable and Object Collection: Implemented to create scrollable lists and collections of
objects, enhancing the organization and accessibility of information.
8. Voice Input: Integrated for hands-free interaction, allowing users to input commands or
queries using voice commands.
9. By incorporating these MRTK functions and widgets, we aimed to create a dynamic and
immersive learning experience within our augmented reality application.
4. MOS Celje Exhibition
In October 2023, we seized the opportunity to present our application at the MOS Exhibition
in the city of Celje. This exhibition stands as the largest in Slovenia, making it crucial for
us to gather feedback, assess the application’s performance in real conditions, and refine
any shortcomings. For a week, our booth welcomed visitors to experience our application
firsthand. This provided us with valuable insights, allowing us to enhance the application
based on real user interactions. Despite receiving overwhelmingly positive feedback, the initial
days prompted us to identify a few areas for improvement in both the application and our
presentation. During the exhibition we understood the importance of introducing the users,
especially for unfamiliar with augmented reality, to the basic control concepts. Explaining how
to interact with buttons, various available gestures, and navigating inside the virtual world
proved crucial. These fundamental aspects, when explained correctly, make application usage
experience better. We addressed these aspects, ensuring that our presentation was clear and
compelling for visitors of various ages.
5. Conclusion
In conclusion, the development resulted in an application designed for the in-depth study of
human anatomy, incorporating all planned features. In this work, various challenges were
encountered, and therefore key solutions were prepared. The application underwent real user
testing, yielding positive feedback and audience interest. As augmented reality technologies
continue to evolve, ongoing refinement remains imperative for future advancements in the
field.
References
[1] T. Little, 21st century learning and progressive education: An intersection, International
Journal of Progressive Education 9 (2013) 84–96.
[2] S. Ong, V. K. Siddaraju, S. Ong, V. K. Siddaraju, Introduction to the mixed reality toolkit,
Beginning Windows Mixed Reality Programming: For HoloLens and Mixed Reality Headsets
(2021) 85–110.
[3] B. Ramic-Brkic, Enhancing progressive education through the use of serious games, in:
2018 10th International Conference on Virtual Worlds and Games for Serious Applications
(VS-Games), IEEE, 2018, pp. 1–4.
[4] A. R. Sadovnik, S. F. Semel, R. W. Coughlan, B. Kanze, A. R. Tyner-Mullings, Progressive
education in the 21st century: The enduring influence of john dewey, The Journal of the
Gilded Age and Progressive Era 16 (2017) 515–530.
[5] J. Garzón, An overview of twenty-five years of augmented reality in education, Multimodal
Technologies and Interaction 5 (2021) 37.
[6] C. Avila-Garzon, J. Bacca-Acosta, J. Duarte, J. Betancourt, et al., Augmented reality in
education: An overview of twenty-five years of research., Contemporary Educational
Technology 13 (2021).
[7] M. Kljun, V. Geroimenko, K. Čopič Pucihar, Augmented reality in education: Current status
and advancement of the field, Augmented Reality in Education: A New Technology for
Teaching and Learning (2020) 3–21.
[8] N. Elmqaddem, Augmented reality and virtual reality in education. myth or reality?,
International journal of emerging technologies in learning 14 (2019).