=Paper=
{{Paper
|id=Vol-3408/short-s1-01
|storemode=property
|title=Human-Robot Collaboration in Healthcare: New Programming and Interaction Techniques
|pdfUrl=https://ceur-ws.org/Vol-3408/short-s1-01.pdf
|volume=Vol-3408
|authors=Luigi Gargioni
|dblpUrl=https://dblp.org/rec/conf/iseud/Gargioni23
}}
==Human-Robot Collaboration in Healthcare: New Programming and Interaction Techniques==
https://ceur-ws.org/Vol-3408/short-s1-01.pdf
Human-robot collaboration in healthcare: new
programming and interaction techniques
Luigi Gargioni1
1
Department of Information Engineering, University of Brescia, Via Branze 38, Brescia, 25123, Italy
Abstract
In the field of healthcare, collaborative robots can be utilized to enhance productivity and efficiency.
This research will delve into the development and investigation of new techniques to enable effective
interaction with collaborative robots. The focus will be on designing innovative techniques that can be
easily understood and implemented by individuals with no technical knowledge in computer science or
robotics. Collaborative robots can be used to automate repetitive tasks, allowing healthcare professionals
to focus on more complex procedures. For instance, by leveraging this technology, it is possible to
significantly increase the efficiency and speed of therapy preparation, ultimately improving patient
outcomes. The study will explore various approaches to simplify and streamline the programming
process, reducing the need for technical knowledge and expertise.
Keywords
Human-Machine Interaction, End-User Development, Human-Robot Collaboration, Collaborative Robots
1. Introduction
The PhD project focuses on the development of empowering technologies for healthcare, in
order to improve and optimize current processes. In particular, novel technologies are needed
to address the continuous rise in the average age of the population [1] and medication errors
that are becoming more and more common [2].
To achieve this goal, collaborative robots can play a relevant role. Collaborative robotics is
one of the most promising areas for innovation in enterprises and processes. In particular, in
the medical sector several repetitive and low value-added tasks of healthcare workers can be
identified, which could be delegated to collaborative robots. Therefore, the ease of use of such
robots and the ability to define tasks by the users are crucial.
Overall, the use of robots in pharmacy is a rapidly growing field, and the technology is
continually improving. As robots become more sophisticated and advanced, they have the
potential to revolutionize the practice of pharmacy, improving patient safety and increasing
efficiency and productivity in the pharmacy.
As a possible final aim of the project, it would be interesting to evaluate whether the techniques
studied for programming collaborative robots can be generalized outside the field of robotics,
such as IoT ecosystem customization and management or No-code development approaches to
information systems.
IS-EUD 2023: 9th International Symposium on End-User Development, 6-8 June 2023, Cagliari, Italy
Envelope-Open luigi.gargioni@unibs.it (L. Gargioni)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
2. Background and related works
This PhD project concerns at least two macro-themes in computer engineering and it is therefore
necessary to specifically analyse the state of the art of each of them. The macro-themes concern
human-computer interaction, specifically end-user development, and robotics, specifically the
subfield of collaborative robots, also known as cobots.
Various approaches to programming and customising software systems by non-computer-
savvy end-users have been proposed in literature. In [3], an analysis of these techniques was
made and organized along three main approaches: end-user development, end-user program-
ming and end-user software engineering. This subdivision is however quite jagged and varies
from community to community, so much so that the concepts expressed by the different terms
often overlap. In the conclusions of the aforementioned article, end-user development (EUD)
is therefore defined as the most general approach encompassing all techniques concerning
the modification, extension, creation and testing of digital artifacts by the end user. From
this synthesis and the numerous sources analysed, it appears that the most commonly used
techniques are the following: those based on components, which are easily implemented thanks
to the concept of modularity in programming, those based on event-condition-action rules, and
programming by examples, while natural language has only a niche sector. The latter type of
approach has been proposed in this context for more than 50 years [4], but it is only recently
that important steps have been taken along this road, although related to specific and restricted
areas. The works described in [5] and [6] consider the application of this type of approach in the
robotics field. The former proposes a method for defining parametric tasks through user-robot
dialogue, while the latter presents a method for translating free natural language sentences into
an instruction sequence for a robot. However, both studies only validate their approaches with
existing datasets, rather than conducting experiments with users. It can be also remarked that
nowadays, the main use of natural language relates to virtual assistants and chatbots. However,
the huge amount of information that can be expressed thanks to the different declinations and
facets of natural language is currently not usable in the world of computing. It is therefore clear
that it has a potential that is still largely untapped.
It can be thought of making full use of this potential of natural language where the device to
be programmed is in close contact, both physically and in terms of interoperability, with the
user and in conditions where agile and unstructured collaboration is required for efficiency and
safety reasons. Collaborative robots fall well within this definition.
As to collaborative robotics, a couple of definitions found in the literature can be taken
into account. In [7] the reason why this technology was conceived is introduced: to be able
to be in close contact with human operators, even outside the classical context of industrial
production lines. For this to happen, it is necessary that the robot communicates with the users,
understands their needs and behaves accordingly. Another interesting definition can be found in
[8]. Here, the importance of safety and new technologies designed to ensure that human-robot
collaboration is functional and safe is emphasised. In other words, these robots must have
devices on board to make it safe for the human to be within their range of action. As far as
both use-time and design-time programming are concerned, this is for now left to robotics and
programming experts. Methodologies have therefore been proposed to simplify programming
through the use of components, i.e. graphically facilitated approaches with modular functions
[9] [10], solutions based on trigger-action personalization rules [11] and multimodal on-the-fly
development system [12]. Nevertheless, agile and simple programming that is within the reach
of non-experts is still far from being achieved.
The healthcare sector poses a daunting challenge to any technological system. Firstly, because
of the sterility and safety standards that must be met within the laboratories. Secondly, because
of the chronic shortage of space that plagues laboratories themselves. In addition, one of the
factors limiting the number of patients that can be treated is the need for highly qualified
medical staff. These professionals spend a large part of the day on repetitive tasks with low
added value. By automating these tasks with the use of robots, healthcare professionals will be
able to concentrate on more valuable and productive tasks, so that more people can receive the
care they need. Three specific medical fields in which collaborative robots can be applied have
been identified in [13]: diagnostics, surgery and rehabilitation. Diagnostics is the only field in
which there are many robotic applications, particularly in orthopaedic specialisations. Indeed,
there are reports of systems that carry out pre-operative mapping of both bone and cartilage
parts. Regarding surgery, simple tasks such as handing of instruments to the surgeon, or more
complex tasks, such as actual operation, can be considered. Cobots can also play an important
role in improving post-operative rehabilitation processes. The best known case at present is
that of the exoskeletons used for rehabilitation following orthopaedic operations.
At the moment, no consistent solutions were reported to help medical workers in pharmacies
to prepare therapies, a task that is crucial and prone to errors. As for now, this task is performed
manually by pharmacists, consuming cognitive resources and time for a low-value-added task.
In this PhD research, the focus will be on studying the use of cobots for packaging therapies by
pharmacists.
3. Reasons for choosing the particular topic and research
objectives
The interaction with collaborative robots needs to be redesigned, both in terms of the physical
component and in terms of programming and use. As far as the physical aspect is concerned,
considerable progress has been made: to the detriment of speed and efficiency, a collaborative
robot is equipped with sensors that guarantee the safety of the people within its range of action,
thus being able to become a collaborator to which the most burdensome, tedious and precision
tasks can be entrusted, while still retaining control of the entire process in human hands.
When it comes to programming and use, a relevant problem arises, which can be trivially
summarised in the following question: how difficult can it be to collaborate with a person
who does not speak the same language as you? This is a major problem in this new branch of
robotics. If collaborative robots are designed to be in close contact with the human operator in
a work environment, they must also be easy to use by users who are not computer or robotics
specialists. It is therefore essential to identify a new approach to the use of collaborative robots,
and one solution that should be considered is the use of natural language.
Not only the interaction with the robot can benefit from the use of a more effective method
of communication, but also the programming of tasks for a collaborative robot. Run-time
interaction is the one that could benefit most for the execution of a truly collaborative task,
with the human as the protagonist and the robot as an active collaborator. Also of interest are
design-time programming, as addressed in [14] and [15], and run-time programming, which
still needs to be further explored and extended by introducing artificial intelligence techniques.
Regarding the purpose of use, cobots in pharmacy has the potential to revolutionize the indus-
try by improving efficiency, accuracy, and safety. In therapy preparation, cobots can automate
many of the labor-intensive and time-consuming tasks involved in medication preparation,
reducing the risk of errors and improving productivity. While the initial cost of cobots may be
high, their long-term benefits can make them a cost-effective solution for pharmacies looking
to improve their operations.
4. Description of the research project
The research project will focus on the study of human-robot collaboration, from different
perspectives: that of direct interaction with the robot during the execution of collaborative
tasks and that of programming new tasks for the robot. For the reasons stated above regarding
the current and future expansion of the use of collaborative robots, it is important to take into
account all the factors present in this human-robot mutual influence. Here are a few key factors
to consider:
• Task suitability: one of the most important factors to consider when collaborating with
robots is task suitability. While robots are great at performing repetitive and routine
tasks, they may not be as adept at handling complex or nuanced tasks that require a
human touch. Therefore, it is important to assess the task at hand and determine whether
a robot is the best fit for the job.
• User interface: the user interface is another important factor to consider when collabo-
rating with robots. The interface needs to be intuitive and easy to use so that humans
can easily interact with the robot. Additionally, it should be designed to encourage
collaboration between the human and the robot. Effective communication is key to a
successful human-robot collaboration. The robot should be able to communicate with the
human and ensure that is clear and easy to understand, and the human should be able to
communicate their needs and expectations to the robot.
• Safety: safety is always a concern when working with robots. It is important to ensure
that the robot is designed with safety in mind, and that appropriate safety measures are
implemented to protect both the human and the robot.
• Training: proper training is essential when working with robots. Humans need to be
trained on how to interact with the robot and how to operate it safely. Additionally,
robots may need to be trained on how to work with humans in order to optimize their
performance.
• Ethics: ethical considerations must be taken into account when working with robots.
For example, questions about ownership and responsibility may arise when a robot is
involved in a task.
At the pharmacy/pharmaceutical distributor level, a robotic module would collaborate with
the operator to package the medications. The system would be installed integrating the workflow
already present in each pharmacy, embedding automatic or semi-automatic technologies.
The main focus of this research is regarding therapy preparation. Once data is retrieved from
currently used prescription systems (quantity and frequency of administration), the therapy
preparation phase is enabled along with the preparation of pillboxes by the pharmacy. The
various therapies are prepared through the workflow described in Figure 1. At this stage, the use
of collaborative robots becomes central. The pharmacist in charge of therapy preparation will
be supported by a cobot to prepare the pillboxes. The pharmacist is then in charge of creating
the tasks for the robot and interacting with it at run time. The pill packaging process is prone
to error by the pharmacist as well as having low added-value. It is therefore important that it
can be automated as much as possible, but always leaving the user, in this case the pharmacist,
at the center of the process. It is equally important to use a collaborative robot for this task
because of the close contact with the operator, but also because of the frequent re-programming
of tasks.
The aim of this study is to delve into different types of programming and interaction tech-
niques with the robot, in order to be able to find a single technique or the combination of
different ones that can fit the successful execution of the task. These techniques will be intercon-
nected to provide the most efficient interaction possible, keeping natural language processing at
the centre. The entire workflow will then be studied in order to build this interaction to ensure
effectiveness and efficiency for both the operator and the process. This is meant to leave the
pharmacists free to deal with tasks where they can generate added-value by their knowledge
in the field and to avoid frustration caused by repetitive tasks, which can lead to errors. Of
course, it could lead to bigger problems if pharmacists find it difficult to programming and use
the robot. It is important that this collaboration is as easy and flexible as possible because most
probably the pharmacist will not be an expert in programming robotic tasks.
Figure 1: The collaborative workflow
5. Research methodology
In order to understand which is the best interaction to propose, different interviews will be
conducted with pharmacists to gather the basic requirements.
User-centred design and/or participatory design through scenarios, personas, static and
dynamic prototypes will be used. It will then be necessary to carry out various experiments
with users, in order to evaluate and have the most truthful feedback possible on the progress of
the development path taken.
The User Experience Questionnaire (UEQ) [16] [17] and the NASA Task Load Index (NASA-TLX)
[18] will be used to investigate the subjective experience and cognitive demands of users while
engaged in the proposed interaction. By leveraging these two instruments, a comprehensive
analysis of the user experience and workload associated with the interaction in question will be
provided.
Another type of questionnaire that will be used to evaluate the proposed solutions will
be the System Usability Scale (SUS) [19]. In addition to administering the questionnaire, it is
also essential to collect qualitative feedback from users. This can be done through interviews,
focus groups, direct observation and think aloud to gain a deeper understanding of the users’
experiences with the system or product.
It might also be interesting to apply more specific metrics for human-robot collaboration,
like Robot Anxiety Scale (RAS) [20] and guidelines for design and evaluation [21].
In conclusion, it is interesting to point out that the project in question is not concerned with
addressing the aforementioned issues in order to give robotics a precise role in the medical field,
but instead it aims to propose reusable approaches to programming and interaction in order to
exploit the potential of cobots for the largest possible variety of tasks in the healthcare domain,
such as those illustrated in the previous examples.
6. Conclusion
The doctoral path started just a few months ago, so at this time there are no tangible results to
report yet. As of now, user interaction is being studied by designing scenarios, personas, and
prototype workflows. The PhD project is oriented towards the in-depth study of therapy prepa-
ration by pharmacists. In this phase they will have to interact and create programms for a cobot,
so a well designed End-User Development technique will be of paramount importance. The
results that will be achieved will be of great interest regarding the introduction of collaborative
robots in the medical field. In addition, as already mentioned, there will always be the goal to
generalize the concepts studied outside this specific context in the future, bringing benefits to
the field of Human-Computer Interaction at large.
Acknowledgments
This project and the relevant PhD scholarship are co-funded by the Italian Ministry1 and the
company Antares Vision2 .
I thank Prof. Daniela Fogli and Prof. Pietro Baroni for the supervision in this PhD program
and Mr. Dario Sala and Dr. Enrico Almici for the management of the project on behalf of
Antares Vision.
References
[1] Eurostat, Ageing Europe - Statistics on population developments, 2020. URL:
https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Ageing_Europe_
-_statistics_on_population_developments.
[2] M. Aseeri, G. Banasser, O. Baduhduh, S. Baksh, N. Ghalibi, Evaluation of medication error
incident reports at a tertiary care hospital, Pharmacy 8 (2020) 69.
[3] B. R. Barricelli, F. Cassano, D. Fogli, A. Piccinno, End-user development, end-user pro-
gramming and end-user software engineering: A systematic mapping study, Journal of
Systems and Software 149 (2019) 101–137.
[4] J. E. Sammet, The use of English as a programming language, Communications of the
ACM 9 (1966) 228–230.
[5] G. Gemignani, E. Bastianelli, D. Nardi, Teaching robots parametrized executable plans
through spoken interaction, in: Proceedings of the 2015 International Conference on
Autonomous Agents and Multiagent Systems, 2015, pp. 851–859.
[6] D. K. Misra, J. Sung, K. Lee, A. Saxena, Tell me Dave: Context-sensitive grounding of
natural language to manipulation instructions, The International Journal of Robotics
Research 35 (2016) 281–300.
[7] L. Rozo, S. Calinon, D. G. Caldwell, P. Jimenez, C. Torras, Learning physical collaborative
robot behaviors from human demonstrations, IEEE Transactions on Robotics 32 (2016)
513–527.
[8] W. Wannasuphoprasit, R. B. Gillespie, J. E. Colgate, M. A. Peshkin, Cobot control, in:
Proceedings of International Conference on Robotics and Automation, volume 4, IEEE,
1997, pp. 3571–3576.
[9] D. Weintrop, A. Afzal, J. Salac, P. Francis, B. Li, D. C. Shepherd, D. Franklin, Evaluating
coblox: A comparative study of robotics programming environments for adult novices, in:
Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018,
pp. 1–12.
[10] J. Huang, M. Cakmak, Code3: A system for end-to-end programming of mobile manipulator
robots for novices and experts, in: Proceedings of the 2017 ACM/IEEE International
Conference on Human-Robot Interaction, 2017, pp. 453–462.
[11] N. Leonardi, M. Manca, F. Paternò, C. Santoro, Trigger-action programming for personal-
1
Piano Nazionale di Ripresa e Resilienza (PNRR): https://www.mef.gov.it/focus/
Il-Piano-Nazionale-di-Ripresa-e-Resilienza-PNRR/
2
Antare Vision S.p.A.: https://www.antaresvisiongroup.com/lifescience/it/home/
ising humanoid robot behaviour, in: Proceedings of the 2019 CHI Conference on Human
Factors in Computing Systems, 2019, pp. 1–13.
[12] D. Porfirio, L. Stegner, M. Cakmak, A. Sauppé, A. Albarghouthi, B. Mutlu, Sketching robot
programs on the fly, arXiv preprint arXiv:2302.03088 (2023).
[13] Homberger, Il ruolo dei cobot nelle applicazioni medicali, 2022. URL: https://www.
homberger-robotica.com/news/il-ruolo-dei-cobot-nelle-applicazioni-medicali/.
[14] S. Beschi, D. Fogli, F. Tampalini, Capirci: a multi-modal system for collaborative robot
programming, in: End-User Development: 7th International Symposium, IS-EUD 2019,
Hatfield, UK, July 10–12, 2019, Proceedings 7, Springer, 2019, pp. 51–66.
[15] D. Fogli, L. Gargioni, G. Guida, F. Tampalini, A hybrid approach to user-oriented pro-
gramming of collaborative robots, Robotics and Computer-Integrated Manufacturing 73
(2022) 102234. URL: https://www.sciencedirect.com/science/article/pii/S073658452100106X.
doi:https://doi.org/10.1016/j.rcim.2021.102234 .
[16] B. Laugwitz, T. Held, M. Schrepp, Construction and evaluation of a user experience
questionnaire, in: HCI and Usability for Education and Work: 4th Symposium of the Work-
group Human-Computer Interaction and Usability Engineering of the Austrian Computer
Society, USAB 2008, Graz, Austria, November 20-21, 2008. Proceedings 4, Springer, 2008,
pp. 63–76.
[17] M. Schrepp, A. Hinderks, J. Thomaschewski, Applying the user experience questionnaire
(ueq) in different evaluation scenarios, in: Design, User Experience, and Usability. Theories,
Methods, and Tools for Designing the User Experience: Third International Conference,
DUXU 2014, Held as Part of HCI International 2014, Heraklion, Crete, Greece, June 22-27,
2014, Proceedings, Part I 3, Springer, 2014, pp. 383–392.
[18] S. G. Hart, Nasa task load index (tlx) (1986).
[19] J. Brooke, et al., Sus-a quick and dirty usability scale, Usability evaluation in industry 189
(1996) 4–7.
[20] T. Nomura, T. Suzuki, T. Kanda, K. Kato, Measurement of anxiety toward robots, in:
ROMAN 2006-The 15th IEEE International Symposium on Robot and Human Interactive
Communication, IEEE, 2006, pp. 372–377.
[21] H. A. Frijns, C. Schmidbauer, Design guidelines for collaborative industrial robot user
interfaces, in: Human-Computer Interaction–INTERACT 2021: 18th IFIP TC 13 Inter-
national Conference, Bari, Italy, August 30–September 3, 2021, Proceedings, Part III 18,
Springer, 2021, pp. 407–427.
About the author
Luigi Gargioni is PhD Student at the Department of Information Engineering, University of
Brescia, Italy. His research interests are Human-Computer Interaction, Human-Robot Collab-
oration, Collaborative Robots and End-User Development. He is part of the Technology for
Health PhD program at the University of Brescia.