=Paper=
{{Paper
|id=Vol-3609/paper27
|storemode=property
|title=Teaching Assistant Robots in Various Fields: Natural Sciences, Medicine and Specific Non-Deterministic Conditions
|pdfUrl=https://ceur-ws.org/Vol-3609/short7.pdf
|volume=Vol-3609
|authors=Roman Hasko,Oleksandra Hasko,Hakan Kutucu
|dblpUrl=https://dblp.org/rec/conf/iddm/HaskoHK23
}}
==Teaching Assistant Robots in Various Fields: Natural Sciences, Medicine and Specific Non-Deterministic Conditions==
https://ceur-ws.org/Vol-3609/short7.pdf
Teaching Assistant Robots in Various Fields: Natural Sciences,
Medicine and Specific Non-Deterministic Conditions
Roman Haskoa, Oleksandra Haskoa, Hakan Kutucub
a
Lviv Polytechnic National University, 12, Stepan Bandera str., Lviv, 79000, Ukraine
b
Karabuk University, Department of Software Engineering, Karabuk, 78050, Turkey
Abstract
The article describes an approach to creating a Robot Assistant as an intelligent system
that uses generative artificial intelligence (GAI) and has a physical body and a voice
interface for communication. To understand the actions of the assistant and increase trust
in the system, Explainable AI is used, and for better efficiency - Federated Learning, that
is, the robot learns on distributed data without the need for centralized collection of this
data. XAI allows users to understand what algorithms and data are driving the decision-
making assistant. Access to Large Language Models means that the robot assistant uses
powerful language models such as GPT-3.5 to understand and generate language. This
allows the assistant to understand user requests and provide information and
recommendations based on a large amount of knowledge and texts.
Robot interactions with the real world are becoming better and more predictable thanks
to Embodied AI. It can perform tasks related to moving physical objects, understand the
gestures and movements of users, and interact with objects in real-time, moving in
different places and environments.
The software solution is based on ROS2 with the necessary extensions for the listed
technologies. All this together makes the work of the assistant effective, providing
understanding and explanation of decisions, protecting data privacy and providing natural
communication through the voice interface.
Keywords 1
Robotics, Federated learning, Explainable AI, XAI, Embodied AI, Telepresence, ROS.
1. Introduction
Assistant robots can be very useful in education in a variety of fields, including science, medicine,
and areas with adverse conditions. They can provide support and assistance to students, teachers and
professionals in these fields. Here are some ways that can be used:
1. Explanation of material: Assistant robots can provide explanations of complex topics using text,
visual or audio materials. In the natural sciences, this can include mathematical calculations, chemical
reactions, and physical laws. In medicine, they can help understand the anatomy, physiology and
treatment of certain diseases.
2. Simulations and Virtual Labs: In the natural sciences, robot assistants can create virtual labs and
simulations that allow students to study experiments and phenomena in a safe environment. In
medicine, they can be used for medical simulation and training.
3. Personalized learning approaches: Assistant robots can adapt learning to the needs of each student.
They can take into account knowledge level, interests and pace of learning to provide effective
individual support.
4. Progress Tracking: Assistant robots can track student progress and provide reports to teachers or
academics. This helps to identify the problems and needs of students in time.
IDDM’2023: 6th International Conference on Informatics & Data-Driven Medicine, November 17 - 19, 2023, Bratislava, Slovakia
EMAIL: r.hasko@gmail.com (A. 1); oleksandra.l.hasko@lpnu.ua (A. 2); hakankutucu@karabuk.edu.tr (A.3)
ORCID: 0000-0001-5923-6577 (A. 1); 0000-0003-4519-610X (A. 2); 0000-0001-7144-7246 (A. 3)
©️ 2023 Copyright for this paper by its authors.
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
5. Access to resources: Assistant robots can provide access to libraries of scientific sources and
publications. In medicine, this may include access to databases of medical journals and clinical
guidelines.
6. Support in dangerous environments: In specific dangerous environments, such as deep space
exploration, underwater operations or emergencies, assistant robots can be used to collect data, monitor
and provide recommendations without risking human life.
7. Providing access to expert opinion: In teaching and research in any field, robot assistants can help
students and researchers gain access to global experts and consultants through virtual conferences and
communication systems.
It is important to note that successful collaboration between assistant robots and humans requires
careful software development, integration with real learning and work processes, and ensuring data
privacy and security. Each specific industry may have its own unique requirements and challenges, and
developers must take these into account.
The world is witnessing the growth of more diverse learning methods, in particular, using robotics,
the Internet of Things (IoT) and augmented and virtual reality (AR/VR) applications. This allows smart
devices to share data over the Internet and improve the quality of learning. Virtual Reality and
Augmented Reality enable the creation of immersive learning environments. For example, Microsoft
HoloLens allows students to interact with virtual objects to teach science subjects.
The use of generative artificial intelligence plays a special role. Successful collaboration between
assistant robots and humans requires careful software development, integration with real-world learning
and work processes, and data privacy and security.
Special cloud web-oriented educational platforms have been developed [1, 2, 3] and a separate class
of software has appeared - e-learning. Now, it's time for specialized hardware for training. Such systems
are mainly built on the basis of a decentralized architecture and use the capabilities of cloud services
for communication mainly through a specialized API, for example, access to ChatGPT from OpenAI.
Due to the predominant focus on the Internet and cloud technologies, such systems are mostly not
very effective when used in the modern hybrid learning process, which requires a quick reaction and
taking into account various obstacles in the communication process.
Certain features of training are present in specific fields, for example, in medical education, modern
blended learning technologies using robotic educational platforms, in particular specialized
telepresence robots for remote participation of students from different geographically distant places
with maximum immersion in the educational process, can be particularly effective [4, 5].
Telepresence robots have been used for a relatively long time and not only in education, so we are
talking about a fundamentally different, new level of their autonomy. Thanks to the use of LLM
capabilities, such robots become real assistants and their remote control capability translates the
efficiency and accessibility of training as well as the practical experience of future specialists to a new
level with better results in their future professional activities.
2. Background
The appearance of assistant robots in education was preceded by a number of stages and
technological developments. Here are some key events and factors that created the basis for the
emergence of this technology:
1. Development of information technology: With the advent of computers and the Internet, access
to a large amount of information and interactive learning has become possible. Information technologies
have become the basis for the development of educational technologies.
2. E-Literature and Learning Platforms: The launch of e-textbooks and a learning platform allows
students to study online and access various learning resources, including video lessons and interactive
tasks.
3. Development of artificial intelligence: The gradual development of natural language processing
and machine learning technologies has made it possible to create robot assistants capable of
understanding and generating human speech. This opened the door to the development of intelligent
learning systems.
4. Scaling of data processing: The increase in the power of computing systems and the increased
availability of large amounts of data have made it possible to create assistant robots that can quickly
analyze information and provide answers.
5. Development of educational initiatives: Interest in improving the quality of education and
accessibility to it has led to the funding and development of innovative educational technologies.
6. Change in learning approach: The learning paradigm is also changing. Today's students and
professionals are more familiar with virtual and online learning, creating a demand for technologies
that meet these needs.
7. Research in the field of education and psychology: The study of learning processes and the
development of cognitive sciences and psychology contributed to the creation of more effective
teaching methods and interactive systems.
All of these factors have contributed to the development of teaching assistant robots that provide
students and learners with access to individualized learning, additional resources, and support that can
enhance the quality of learning and facilitate the effective acquisition of new knowledge.
3. Related works
Current developments in robotics to assist in the educational process include a wide range of
innovative technologies and robots that contribute to improving the quality and accessibility of
education. Robotics in education offers an impressive range of innovative solutions that facilitate
learning and increase its effectiveness.
Robot-assistants for training are primarily AI-based personal assistants. By using artificial
intelligence like GPT-3, they can provide personalized support to students, explain complex concepts
and answer questions. Related developments should also be included here - social robots like Pepper,
which can be used to interact with people with various special needs, promoting social integration and
learning communication skills. At the same time, the use of multi-agent robots to create educational
games allows for the development of cooperation, communication and problem-solving skills.
This entire successful development process is hard to imagine without virtual and augmented reality.
Thanks to these technologies, simulators and virtual laboratories have become possible, which allow
students to study complex phenomena and experiments safely. And augmented reality, for example in
applications for smartphones and AR-glasses, provides additional information and an interactive
experience in the learning process.
Special mention should be made of exoskeletons for medical education, which help students and
medical staff learn anatomy, perform surgeries and research new treatments. Robots for medical
education with simulators allow medical students to practice performing operations and diagnosing
patients without risk to life [6].
AI support for teachers. Platforms that use artificial intelligence to analyze learning data and create
personalized plans for students can make the work of teachers easier and improve the quality of
learning. AI-powered learning platforms like Coursera, edX, and Udacity use machine learning
algorithms to personalize learning and predict student performance.
Assistant robots, such as NAO and Pepper from SoftBank, as well as robots from Boston Dynamics,
can be used to teach various educational subjects. They can provide interactive lessons using speech
and facial reproduction, which helps create positive interactions with students.
Some works are designed to improve students' speaking skills. For example, robots from the
company Leka are used to teach children to interact and develop speech skills, and companion robots
for students with special needs, such as QTrobot, can be used to teach and develop children with specific
needs.
Special mention should be made of robots for STEM education [7], which specialize in teaching
scientific subjects such as mathematics, programming and robotics. Robots such as Wonder Workshop's
Dash and Dot help create interactive activities for children in these areas.
These are just a few examples of current developments in the field of robotics for education. With
the development of technologies and innovations, further expansion of opportunities for enriching the
educational process and increasing interest in learning among students is expected. Modern
developments in robotics for learning are actively developing and help improve learning processes and
provide more accessible and interactive education. Such robots and technologies expand learning
opportunities and contribute to the development of digital education [8].
4. Proposed Approach
The proposed approach is based on the previous experience of creating autonomous learning robots
and takes into account the latest achievements in AI, in particular generative AI and LLM. The design
is a fairly autonomous robotic platform consisting of the following main parts:
1. a mobile wheeled chassis with a system of motors, a control controller (for example, Arduino or
STM32), a power controller with batteries and a line of sensors in the composition:
- ultrasonic sensors with an emitter
- infrared sensors with an emitter
- radio frequency module with Bluetooth support
2. a smart unit based on the Nvidia Jetson microcomputer with a camera and computer vision
support. Previous versions used a Raspberry Pi with a separate Intel Movidius to run the pre-trained
neural networks, but the switch to Nvidia's platform looks more promising both in terms of efficiency
and power, as well as a wide selection of specialized robotics software.
3. a user interface with an emphasis on voice input and output and additional capabilities such as a
display and manipulators based on the robot arm. The feature of this voice interface is its versatility,
autonomy and high degree of "intelligence" thanks to three components:
- voice recognition with conversion to text
- connection with OpenAI API (ChatGPT) in the format of exchanging text messages
- voice synthesis based on text received from ChatGPT.
In addition, the expansion of the possibilities of the user interface is provided due to the analysis of
both the text and the position of the face, in particular facial expressions of the user. Thanks to this, the
quality of human-machine interaction is moving to a qualitatively better level. More details about the
use of ChatGPT in robotics are shown in Figure 1 by He, Hongmei [9]
Figure 1: Framework of RobotGPT. Copyright by He, Hongmei. (2023). [9]
The software of the described robotic platform is based on ROS2 - Robot Operating System of the
second generation. At the same time, a number of features should be noted:
1. Use of generative AI (LLM) on the example of API from OpenAI, namely ChatGPT 3.5 with a
two-way voice interface and a number of additional features.
2. Federated learning, thanks to which it is possible to increase the autonomy of the robot and its
operation in the presence of obstacles and interruptions in connection with the cloud service. Thanks to
both the capabilities of ROS and decentralized machine learning, it is possible to ensure sufficiently
"intelligent" behavior of the robot in the absence of commands from the operator.
3. Explainable AI and Cooperation of Robots. This is another set of very important features of the
described robot, as it allows a human to "understand" the behavior of the robot, through clarifying
commands and questions with answers actually to program and improve the behavior of the robot. In
turn, cooperation between individual robots, their joint interaction brings their efficiency to a
qualitatively new level and allows teams of robots and people to cooperate.
4.1. Peculiarities of using generative AI (LLM) in the educational
process
LLMs (Large Language Models) such as GPT-3 open many possibilities for the development of
educational technologies. They play key roles in the following aspects such as рersonalized learning.
LLMs can create individualized learning materials based on each student's needs and skill level. They
can approach each student individually and provide additional explanations or tasks to strengthen their
understanding of the material.
The second option is teacher and instructor support. LLMs can help teachers create curricula,
materials, and tests. They can also make recommendations for improving teaching and evaluating
student performance.
The next option is content generation automation, question answers and explanations. LLMs can
generate text and visual content for learning materials, including textbooks, articles and how-tos, and
provide detailed explanations and answers to questions about various subjects and topics. This can
significantly facilitate the development of learning resources and is useful for students looking for
additional information or clarification on specific issues.
Another option of using LLM is instant access to information. Robot assistants can quickly provide
information on various subjects and topics, which helps students effectively search and find answers to
their questions.
Use in interactive applications. LLMs can be used in web services and learning applications where
they can interact with users via text chats or voice commands. In higher education and research, LLMs
can be used to quickly access a large number of scholarly articles, data, and information to support
research.
However, it is important to remember that the use of the LLM also involves ethical issues relating
to authorship, data sources and transparency. Also, the development of these technologies requires
attention to security and privacy issues in the educational context.
4.2. Federated learning and Embodied AI
Federated or collaborative learning is a specific decentralized approach to training machine learning
models that does not require the exchange of data between cloud services and individual robots or other
client devices. Instead, the raw data on the autonomous robots is used to train the model locally. This
increases the autonomy and confidentiality of data.
Figure 2: Federated learning framework. Copyright by Huang, Xixi [12]
Federated learning approach in medical education can be particularly useful because of the
confidentiality and security of medical data. The main features of federated learning in the medical field
include data decentralization, where instead of collecting all medical data in one place, federated
learning allows the data to be kept on local servers or devices such as medical equipment, smartphones
or desktop PCs. The model is trained locally on each device and only aggregated parameters are sent to
a centralized server, which also helps reduce the risk of leaking sensitive medical data because the data
itself does not leave local servers or devices. Federated learning in medical education can be used to
create and improve models for medical diagnosis, image processing, patient data analysis, and many
other tasks while maintaining the privacy and security of medical data.
Embodied AI [6] combines computational intelligence with the physical body of a robot or agent.
This approach enables robots to interact with the physical world and environment, making them ideal
for use as assistants or helpers. The structure of embodied AI is aimed at solving problems of perception,
planning and execution of actions in real time. The combination of embodied AI and federated learning
can be useful in the development of assistant robots, as it allows improving the quality of interaction
and robot learning. Several assistant robots can learn on different devices, and then combine their
knowledge in one centralized model using federated learning. The use of federated learning helps to
store user data on their devices, which allows for a higher level of privacy. Assistant robots equipped
with embodied AI can work with these models without disclosing users' personal information.
4.3. Explainable AI and Cooperation of Robots
The proposed approach based on the ROS2 [13] operating system of robots together with the use of
generative AI, described software architecture, support for federated learning, explainable AI and
communication capabilities between robots for control and their interaction allows the simultaneous
use of several personalized "smart" assistant robots. In addition, if necessary, other robotic components
are involved in the educational process, for example, manipulators based on robot hands.
Explainable AI (XAI) is an approach to creating intelligent systems that are able to explain their
decisions and actions in a human-understandable form. This approach is particularly important in
robotics, as robots can interact with humans in a variety of domains, and explaining their actions can
be critical for safety, trust, and acceptance. Explainable AI in robotics involves clearly explaining
decisions. Robots, thanks to XAI, try to reveal how they came to a certain decision or action. This may
include displaying the logic, calculations and data used to make decisions. XAI can provide visual
representations of its decisions and processes occurring inside the robot. This can be graphical charts,
diagrams or other visual tools.
Figure 3: Example of cooperation between a team of robots and people. Copyright by R.Hasko [3]
In this example, the interaction of several robots with people in a team. Individual assistant robots
interact with each other using available communication channels such as radio, infrared or ultrasonic
and with people mainly through a voice interface.
5. Results
The proposed approach makes it possible to create qualitatively better assistant robots for various
fields of use and to integrate them into existing or new educational systems. The modular approach
allows you to adapt the system to specific needs.
The emergence of LLM has opened up new opportunities for AI, and its use in the described assistant
robot opens up completely new opportunities compared to previous versions of robots. At the same
time, the emphasis on the use of a "smart" voice interface significantly improves human-machine
interaction.
The use of federated learning and specialized AI-oriented microcomputers ensures a high level of
autonomy and independence for such a robot.
Thanks to the use of XAI, the robot can be applied in various fields, in particular in medicine to
explain diagnoses and treatments and as an autonomous moving platform with explanations of its
behavior and in many other fields. This approach is essential to ensure safety and trust in robots in
various human interaction scenarios.
Taking into account the possibilities of interaction between individual robots for their teamwork
makes it possible to effectively use the described robotic platform both in the educational process and
in many other fields.
6. Conclusion and Future Directions
In this article, we proposed a modern approach to the creation of highly mobile intelligent assistant
robots as a synergy of robotics with modern artificial intelligence technologies and a convenient user
interface with an emphasis on voice input-output as the most convenient for humans.
7. References
[1] Hasko, R. et al. MEdOps - Medical Education with Emphasis on Robotics. IDDM-2022: 5th
International Conference on Informatics & Data-Driven Medicine, November 18–20, 2022,
Lyon, France. CEUR Workshop Proceedings. – 2022. – Vol. 3302 : P. 266–271.
[2] Josie Hughes at al. Embodied Artificial Intelligence: Enabling the Next Intelligence
Revolution. 2022 IOP Conf. Ser.: Mater. Sci. Eng. 1261 012001
[3] Hasko, R., Shakhovska, N. Tripled Learning: Conception and First Steps. CEUR Workshop
Proceedings, 2018, 2105: 481–484.
[4] A. Latip, Y. Andriani, S. Purnamasari and D. Abdurrahman. Integration of educational robotic
in STEM learning to promote students' collaborative skill. Journal of Physics: Conference
Series, Volume 1663, 5th Seminar Nasional Matematika dan Pendidikan Matematika
(SENATIK) 2020 12-13 August 2020, Semarang, Indonesia
[5] Hasko, R. et al. A mixed Fog/Edge/AIoT/robotics education approach based on tripled
learning[J]. CEUR Workshop Proceedings, 2020, 2616: 227–236.
[6] Jiafei Duan et al. A Survey of Embodied AI: From Simulators to Research Tasks.
doi:10.48550/arXiv.2103.04918
[7] Shakhovska, N., et. al. The Student Training System Based on the Approaches of
Gamification[J]. Advances in Intelligent Systems and Computing, 2020, 938: 579–589.
[8] Shakhovska, N., et. al. The method of big data processing for distance educational system.
Advances in Intelligent Systems and Computing, 2018, 689, pp. 461–473
[9] Darmawansah, D., Hwang, GJ., Chen, MR.A. et al. Trends and research foci of robotics-based
STEM education: a systematic review from diverse angles based on the technology-based
learning model. IJ STEM Ed 10, 12 (2023). https://doi.org/10.1186/s40594-023-00400-3
[10] Oña Edwin Daniel; et. al.. "Robotics in health care: Perspectives of robot-aided
interventions in clinical practice for rehabilitation of upper limbs". Applied Sciences. 2019, 9
(13): 2586. doi:10.3390/app9132586. ISSN 2076-3417.
[11] He, Hongmei. (2023). RobotGPT: From ChatGPT to Robot Intelligence.
10.36227/techrxiv.22569247.
[12] Huang, Xixi & Ding, Ye & Jiang, Zoe & Qi, Shuhan & Wang, Xuan & Liao, Qing.
(2020). DP-FL: a novel differentially private federated learning framework for the unbalanced
data. World Wide Web. 23. 10.1007/s11280-020-00780-4.
[13] Steve Macenski at al. Robot Operating System 2: Design, Architecture, and Uses In
The Wild. https://doi.org/10.48550/arXiv.2211.07752