Human-robot collaboration in healthcare: new program ming and interaction techniques

Abstract

In the field of healthcare, collaborative robots can be utilized to enhance productivity and eficiency. This research will delve into the development and investigation of new techniques to enable efective interaction with collaborative robots. The focus will be on designing innovative techniques that can be easily understood and implemented by individuals with no technical knowledge in computer science or robotics. Collaborative robots can be used to automate repetitive tasks, allowing healthcare professionals to focus on more complex procedures. For instance, by leveraging this technology, it is possible to significantly increase the eficiency and speed of therapy preparation, ultimately improving patient outcomes. The study will explore various approaches to simplify and streamline the programming process, reducing the need for technical knowledge and expertise.

1. Introduction

The PhD project focuses on the development of empowering technologies for healthcare, in order to improve and optimize current processes. In particular, novel technologies are needed to address the continuous rise in the average age of the population [ 1 ] and medication errors that are becoming more and more common [ 2 ].

To achieve this goal, collaborative robots can play a relevant role. Collaborative robotics is one of the most promising areas for innovation in enterprises and processes. In particular, in the medical sector several repetitive and low value-added tasks of healthcare workers can be identified, which could be delegated to collaborative robots. Therefore, the ease of use of such robots and the ability to define tasks by the users are crucial.

Overall, the use of robots in pharmacy is a rapidly growing field, and the technology is continually improving. As robots become more sophisticated and advanced, they have the potential to revolutionize the practice of pharmacy, improving patient safety and increasing eficiency and productivity in the pharmacy.

As a possible final aim of the project, it would be interesting to evaluate whether the techniques studied for programming collaborative robots can be generalized outside the field of robotics, such as IoT ecosystem customization and management or No-code development approaches to information systems.

2. Background and related works

This PhD project concerns at least two macro-themes in computer engineering and it is therefore necessary to specifically analyse the state of the art of each of them. The macro-themes concern human-computer interaction, specifically end-user development, and robotics, specifically the subfield of collaborative robots, also known as cobots.

Various approaches to programming and customising software systems by non-computersavvy end-users have been proposed in literature. In [ 3 ], an analysis of these techniques was made and organized along three main approaches: end-user development, end-user programming and end-user software engineering. This subdivision is however quite jagged and varies from community to community, so much so that the concepts expressed by the diferent terms often overlap. In the conclusions of the aforementioned article, end-user development (EUD) is therefore defined as the most general approach encompassing all techniques concerning the modification, extension, creation and testing of digital artifacts by the end user. From this synthesis and the numerous sources analysed, it appears that the most commonly used techniques are the following: those based on components, which are easily implemented thanks to the concept of modularity in programming, those based on event-condition-action rules, and programming by examples, while natural language has only a niche sector. The latter type of approach has been proposed in this context for more than 50 years [ 4 ], but it is only recently that important steps have been taken along this road, although related to specific and restricted areas. The works described in [ 5 ] and [ 6 ] consider the application of this type of approach in the robotics field. The former proposes a method for defining parametric tasks through user-robot dialogue, while the latter presents a method for translating free natural language sentences into an instruction sequence for a robot. However, both studies only validate their approaches with existing datasets, rather than conducting experiments with users. It can be also remarked that nowadays, the main use of natural language relates to virtual assistants and chatbots. However, the huge amount of information that can be expressed thanks to the diferent declinations and facets of natural language is currently not usable in the world of computing. It is therefore clear that it has a potential that is still largely untapped.

It can be thought of making full use of this potential of natural language where the device to be programmed is in close contact, both physically and in terms of interoperability, with the user and in conditions where agile and unstructured collaboration is required for eficiency and safety reasons. Collaborative robots fall well within this definition.

As to collaborative robotics, a couple of definitions found in the literature can be taken into account. In [ 7 ] the reason why this technology was conceived is introduced: to be able to be in close contact with human operators, even outside the classical context of industrial production lines. For this to happen, it is necessary that the robot communicates with the users, understands their needs and behaves accordingly. Another interesting definition can be found in [ 8 ]. Here, the importance of safety and new technologies designed to ensure that human-robot collaboration is functional and safe is emphasised. In other words, these robots must have devices on board to make it safe for the human to be within their range of action. As far as both use-time and design-time programming are concerned, this is for now left to robotics and programming experts. Methodologies have therefore been proposed to simplify programming through the use of components, i.e. graphically facilitated approaches with modular functions [ 9 ] [ 10 ], solutions based on trigger-action personalization rules [ 11 ] and multimodal on-the-fly development system [ 12 ]. Nevertheless, agile and simple programming that is within the reach of non-experts is still far from being achieved.

The healthcare sector poses a daunting challenge to any technological system. Firstly, because of the sterility and safety standards that must be met within the laboratories. Secondly, because of the chronic shortage of space that plagues laboratories themselves. In addition, one of the factors limiting the number of patients that can be treated is the need for highly qualified medical staf. These professionals spend a large part of the day on repetitive tasks with low added value. By automating these tasks with the use of robots, healthcare professionals will be able to concentrate on more valuable and productive tasks, so that more people can receive the care they need. Three specific medical fields in which collaborative robots can be applied have been identified in [ 13 ]: diagnostics, surgery and rehabilitation. Diagnostics is the only field in which there are many robotic applications, particularly in orthopaedic specialisations. Indeed, there are reports of systems that carry out pre-operative mapping of both bone and cartilage parts. Regarding surgery, simple tasks such as handing of instruments to the surgeon, or more complex tasks, such as actual operation, can be considered. Cobots can also play an important role in improving post-operative rehabilitation processes. The best known case at present is that of the exoskeletons used for rehabilitation following orthopaedic operations.

At the moment, no consistent solutions were reported to help medical workers in pharmacies to prepare therapies, a task that is crucial and prone to errors. As for now, this task is performed manually by pharmacists, consuming cognitive resources and time for a low-value-added task. In this PhD research, the focus will be on studying the use of cobots for packaging therapies by pharmacists.

3. Reasons for choosing the particular topic and research objectives

The interaction with collaborative robots needs to be redesigned, both in terms of the physical component and in terms of programming and use. As far as the physical aspect is concerned, considerable progress has been made: to the detriment of speed and eficiency, a collaborative robot is equipped with sensors that guarantee the safety of the people within its range of action, thus being able to become a collaborator to which the most burdensome, tedious and precision tasks can be entrusted, while still retaining control of the entire process in human hands.

When it comes to programming and use, a relevant problem arises, which can be trivially summarised in the following question: how dificult can it be to collaborate with a person who does not speak the same language as you? This is a major problem in this new branch of robotics. If collaborative robots are designed to be in close contact with the human operator in a work environment, they must also be easy to use by users who are not computer or robotics specialists. It is therefore essential to identify a new approach to the use of collaborative robots, and one solution that should be considered is the use of natural language.

Not only the interaction with the robot can benefit from the use of a more efective method of communication, but also the programming of tasks for a collaborative robot. Run-time interaction is the one that could benefit most for the execution of a truly collaborative task, with the human as the protagonist and the robot as an active collaborator. Also of interest are design-time programming, as addressed in [ 14 ] and [ 15 ], and run-time programming, which still needs to be further explored and extended by introducing artificial intelligence techniques.

Regarding the purpose of use, cobots in pharmacy has the potential to revolutionize the industry by improving eficiency, accuracy, and safety. In therapy preparation, cobots can automate many of the labor-intensive and time-consuming tasks involved in medication preparation, reducing the risk of errors and improving productivity. While the initial cost of cobots may be high, their long-term benefits can make them a cost-efective solution for pharmacies looking to improve their operations.

4. Description of the research project

The research project will focus on the study of human-robot collaboration, from diferent perspectives: that of direct interaction with the robot during the execution of collaborative tasks and that of programming new tasks for the robot. For the reasons stated above regarding the current and future expansion of the use of collaborative robots, it is important to take into account all the factors present in this human-robot mutual influence. Here are a few key factors to consider: • Task suitability: one of the most important factors to consider when collaborating with robots is task suitability. While robots are great at performing repetitive and routine tasks, they may not be as adept at handling complex or nuanced tasks that require a human touch. Therefore, it is important to assess the task at hand and determine whether a robot is the best fit for the job. • User interface: the user interface is another important factor to consider when collaborating with robots. The interface needs to be intuitive and easy to use so that humans can easily interact with the robot. Additionally, it should be designed to encourage collaboration between the human and the robot. Efective communication is key to a successful human-robot collaboration. The robot should be able to communicate with the human and ensure that is clear and easy to understand, and the human should be able to communicate their needs and expectations to the robot. • Safety: safety is always a concern when working with robots. It is important to ensure that the robot is designed with safety in mind, and that appropriate safety measures are implemented to protect both the human and the robot. • Training: proper training is essential when working with robots. Humans need to be trained on how to interact with the robot and how to operate it safely. Additionally, robots may need to be trained on how to work with humans in order to optimize their performance. • Ethics: ethical considerations must be taken into account when working with robots.

For example, questions about ownership and responsibility may arise when a robot is involved in a task.

At the pharmacy/pharmaceutical distributor level, a robotic module would collaborate with the operator to package the medications. The system would be installed integrating the workflow already present in each pharmacy, embedding automatic or semi-automatic technologies.

The main focus of this research is regarding therapy preparation. Once data is retrieved from currently used prescription systems (quantity and frequency of administration), the therapy preparation phase is enabled along with the preparation of pillboxes by the pharmacy. The various therapies are prepared through the workflow described in Figure 1. At this stage, the use of collaborative robots becomes central. The pharmacist in charge of therapy preparation will be supported by a cobot to prepare the pillboxes. The pharmacist is then in charge of creating the tasks for the robot and interacting with it at run time. The pill packaging process is prone to error by the pharmacist as well as having low added-value. It is therefore important that it can be automated as much as possible, but always leaving the user, in this case the pharmacist, at the center of the process. It is equally important to use a collaborative robot for this task because of the close contact with the operator, but also because of the frequent re-programming of tasks.

The aim of this study is to delve into diferent types of programming and interaction techniques with the robot, in order to be able to find a single technique or the combination of diferent ones that can fit the successful execution of the task. These techniques will be interconnected to provide the most eficient interaction possible, keeping natural language processing at the centre. The entire workflow will then be studied in order to build this interaction to ensure efectiveness and eficiency for both the operator and the process. This is meant to leave the pharmacists free to deal with tasks where they can generate added-value by their knowledge in the field and to avoid frustration caused by repetitive tasks, which can lead to errors. Of course, it could lead to bigger problems if pharmacists find it dificult to programming and use the robot. It is important that this collaboration is as easy and flexible as possible because most probably the pharmacist will not be an expert in programming robotic tasks.

5. Research methodology

In order to understand which is the best interaction to propose, diferent interviews will be conducted with pharmacists to gather the basic requirements.

User-centred design and/or participatory design through scenarios, personas, static and dynamic prototypes will be used. It will then be necessary to carry out various experiments with users, in order to evaluate and have the most truthful feedback possible on the progress of the development path taken.

The User Experience Questionnaire (UEQ) [ 16 ] [ 17 ] and the NASA Task Load Index (NASA-TLX) [ 18 ] will be used to investigate the subjective experience and cognitive demands of users while engaged in the proposed interaction. By leveraging these two instruments, a comprehensive analysis of the user experience and workload associated with the interaction in question will be provided.

Another type of questionnaire that will be used to evaluate the proposed solutions will be the System Usability Scale (SUS) [ 19 ]. In addition to administering the questionnaire, it is also essential to collect qualitative feedback from users. This can be done through interviews, focus groups, direct observation and think aloud to gain a deeper understanding of the users’ experiences with the system or product.

It might also be interesting to apply more specific metrics for human-robot collaboration, like Robot Anxiety Scale (RAS) [ 20 ] and guidelines for design and evaluation [ 21 ].

In conclusion, it is interesting to point out that the project in question is not concerned with addressing the aforementioned issues in order to give robotics a precise role in the medical field, but instead it aims to propose reusable approaches to programming and interaction in order to exploit the potential of cobots for the largest possible variety of tasks in the healthcare domain, such as those illustrated in the previous examples.

6. Conclusion

The doctoral path started just a few months ago, so at this time there are no tangible results to report yet. As of now, user interaction is being studied by designing scenarios, personas, and prototype workflows. The PhD project is oriented towards the in-depth study of therapy preparation by pharmacists. In this phase they will have to interact and create programms for a cobot, so a well designed End-User Development technique will be of paramount importance. The results that will be achieved will be of great interest regarding the introduction of collaborative robots in the medical field. In addition, as already mentioned, there will always be the goal to generalize the concepts studied outside this specific context in the future, bringing benefits to the field of Human-Computer Interaction at large.

Acknowledgments

This project and the relevant PhD scholarship are co-funded by the Italian Ministry1 and the company Antares Vision2.

I thank Prof. Daniela Fogli and Prof. Pietro Baroni for the supervision in this PhD program and Mr. Dario Sala and Dr. Enrico Almici for the management of the project on behalf of Antares Vision. https://www.mef.gov.it/focus/

References

[1] Eurostat , Ageing Europe - Statistics on population developments, 2020 . URL: https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Ageing_Europe_ -_statistics_on_population_developments.

[2] M. Aseeri , G. Banasser , O. Baduhduh , S. Baksh , N. Ghalibi , Evaluation of medication error incident reports at a tertiary care hospital , Pharmacy 8 ( 2020 ) 69 .

[3] B. R. Barricelli , F. Cassano , D. Fogli , A. Piccinno , End-user development, end-user programming and end-user software engineering: A systematic mapping study , Journal of Systems and Software 149 ( 2019 ) 101 - 137 .

[4] J. E. Sammet, The use of English as a programming language , Communications of the ACM 9 ( 1966 ) 228 - 230 .

[5] G. Gemignani , E. Bastianelli , D. Nardi , Teaching robots parametrized executable plans through spoken interaction , in: Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems , 2015 , pp. 851 - 859 .

[6] D. K. Misra , J. Sung , K. Lee , A. Saxena , Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions , The International Journal of Robotics Research 35 ( 2016 ) 281 - 300 .

[7] L. Rozo , S. Calinon , D. G. Caldwell , P. Jimenez , C. Torras , Learning physical collaborative robot behaviors from human demonstrations , IEEE Transactions on Robotics 32 ( 2016 ) 513 - 527 .

[8] W. Wannasuphoprasit , R. B. Gillespie , J. E. Colgate , M. A. Peshkin , Cobot control, in: Proceedings of International Conference on Robotics and Automation , volume 4 , IEEE, 1997 , pp. 3571 - 3576 .

[9] D. Weintrop , A. Afzal , J. Salac , P. Francis , B. Li , D. C. Shepherd , D. Franklin , Evaluating coblox: A comparative study of robotics programming environments for adult novices , in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems , 2018 , pp. 1 - 12 .

[10] J. Huang , M. Cakmak, Code3: A system for end-to-end programming of mobile manipulator robots for novices and experts , in: Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction , 2017 , pp. 453 - 462 .

[11] N. Leonardi , M. Manca , F. Paternò , C. Santoro , Trigger-action programming for personal1Piano Nazionale di Ripresa e Resilienza (PNRR): Il-Piano-Nazionale-di-Ripresa-e-Resilienza-PNRR/ 2Antare Vision S .p.A.: https://www.antaresvisiongroup.com/lifescience/it/home/ ising humanoid robot behaviour, in: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems , 2019 , pp. 1 - 13 .

[12] D. Porfirio , L. Stegner , M. Cakmak , A. Sauppé , A. Albarghouthi , B. Mutlu , Sketching robot programs on the fly , arXiv preprint arXiv:2302.03088 ( 2023 ).

[13] Homberger , Il ruolo dei cobot nelle applicazioni medicali , 2022 . URL: https://www. homberger -robotica.com/news/il-ruolo-dei-cobot-nelle-applicazioni-medicali/.

[14] S. Beschi , D. Fogli , F. Tampalini , Capirci: a multi-modal system for collaborative robot programming , in: End-User Development: 7th International Symposium , IS-EUD 2019 , Hatfield , UK , July 10-12 , 2019 , Proceedings 7, Springer, 2019 , pp. 51 - 66 .

[15] D. Fogli , L. Gargioni , G. Guida , F. Tampalini , A hybrid approach to user-oriented programming of collaborative robots , Robotics and Computer-Integrated Manufacturing 73 ( 2022 ) 102234 . URL: https://www.sciencedirect.com/science/article/pii/S073658452100106X. doi:https://doi.org/10.1016/j.rcim. 2021 . 102234 .

[16] B. Laugwitz , T. Held , M. Schrepp , Construction and evaluation of a user experience questionnaire, in: HCI and Usability for Education and Work: 4th Symposium of the Workgroup Human-Computer Interaction and Usability Engineering of the Austrian Computer Society , USAB 2008, Graz, Austria, November 20-21 , 2008 . Proceedings 4, Springer, 2008 , pp. 63 - 76 .

[17] M. Schrepp , A. Hinderks , J. Thomaschewski , Applying the user experience questionnaire (ueq) in diferent evaluation scenarios, in: Design, User Experience, and Usability . Theories, Methods, and Tools for Designing the User Experience: Third International Conference , DUXU 2014, Held as Part of HCI International 2014 , Heraklion, Crete, Greece, June 22-27, 2014 , Proceedings, Part I 3 , Springer, 2014 , pp. 383 - 392 .

[18] S. G. Hart , Nasa task load index (tlx) ( 1986 ).

[19] J. Brooke , et al., Sus-a quick and dirty usability scale, Usability evaluation in industry 189 ( 1996 ) 4 - 7 .

[20] T. Nomura , T. Suzuki , T. Kanda , K. Kato , Measurement of anxiety toward robots , in: ROMAN 2006-The 15th IEEE International Symposium on Robot and Human Interactive Communication , IEEE, 2006 , pp. 372 - 377 .

[21] H. A. Frijns , C. Schmidbauer , Design guidelines for collaborative industrial robot user interfaces , in: Human-Computer Interaction-INTERACT 2021 : 18th IFIP TC 13 International Conference , Bari, Italy, August 30-September 3 , 2021 , Proceedings, Part III 18 , Springer, 2021 , pp. 407 - 427 .