=Paper=
{{Paper
|id=Vol-3618/forum_paper_2
|storemode=property
|title=Towards a conceptual safety planning framework for human-robot collaboration
|pdfUrl=https://ceur-ws.org/Vol-3618/forum_paper_2.pdf
|volume=Vol-3618
|authors=Fabian Schirmer,Philipp Kranz,Meenakshi Manjunath,Jeshwitha Jesus Raja,Chad G. Rose,Tobias Kaupp,Marian Daun
|dblpUrl=https://dblp.org/rec/conf/er/SchirmerKMRRKD23
}}
==Towards a conceptual safety planning framework for human-robot collaboration==
https://ceur-ws.org/Vol-3618/forum_paper_2.pdf
Towards a conceptual safety planning framework for
human-robot collaboration
Fabian Schirmer1,2 , Philipp Kranz1 , Meenakshi Manjunath1 , Jeshwitha Jesus Raja1 ,
Chad G. Rose2 , Tobias Kaupp1 and Marian Daun1
1
Center Robotics, Technical University of Applied Sciences Würzburg-Schweinfurt, Schweinfurt, Germany
2
Department of Mechanical Engineering, Auburn University, Auburn, Alabama, USA
Abstract
In human-robot collaboration (HRC), robots interact physically with humans in a shared environment,
without the typical barriers or protective cages used in traditional robotics systems, it is therefore
essential to consider safety measures as a substantial part of the assembly sequence planning (ASP). In
HRC, this process is complicated and time-consuming, especially when ensuring that no safety hazards
are overlooked. This paper reports on first results showing a concept on how to semi-automate the
process of safety analysis for human-robot ASP. Our concept integrates transformer-based natural
language processing NLP models to automatically generate suggestions about potential safety hazards,
their causes and consequences for each assembly step. A human safety expert reviews the pre-populated
information and incorporates supplementary safety considerations via a dashboard. Preliminary results
indicate a significant reduction in manual effort for the expert in the creation of safety measures.
Keywords
Human-Robot Collaboration, Assembly Sequence Planning, Safety Analysis
1. Introduction
Human-robot collaboration (HRC) is a research field with a wide range of applications, future
scenarios, and potentially a high economic impact [1]. Unlike the current factory schema,
where humans work alongside automated robots, in these collaborations, humans engage in
task-sharing with robots, working together on the same objectives. Assembly line robots
assist human workers in tasks like precision welding or intricate component placement. In
this scenario, robots handle the repetitive and intricate tasks, while human workers provide
oversight, quality and decision-making expertise [2].
While traditionally many safety risks were eliminated by a clear separation of working areas
for humans and robots (e.g., by a protective fence) [3], significantly more potential safety risks
have to be considered in a collaborative interaction.
ER2023: Companion Proceedings of the 42nd International Conference on Conceptual Modeling: ER Forum, 7th SCME,
Project Exhibitions, Posters and Demos, and Doctoral Consortium, November 06-09, 2023, Lisbon, Portugal
Envelope-Open fabian.schirmer@thws.de (F. Schirmer); philipp.kranz@thws.de (P. Kranz);
meenakshi.manjunath@study.thws.de (M. Manjunath); jeshwitha.jesusraja@study.thws.de (J. Jesus Raja);
chadgrose@auburn.edu (C. G. Rose); tobias.kaupp@thws.de (T. Kaupp); marian.daun@thws.de (M. Daun)
Orcid 0000-0002-7032-8242 (F. Schirmer); 0000-0002-1057-4273 (P. Kranz); 0009-0005-6421-1450 (M. Manjunath);
0009-0008-7886-7081 (J. Jesus Raja); 0000-0002-8616-393X (C. G. Rose); 0000-0003-3017-5816 (T. Kaupp);
0000-0002-9156-9731 (M. Daun)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Therefore, risk assessment and safety hazard identification are essential steps in HRC assembly
sequence planning (ASP). The ASP process must include a comprehensive risk assessment to
identify potential safety hazards arising from human-robot interactions, equipment malfunction,
or task allocation errors. The challenge lies in accurately identifying all possible safety hazards,
their causes and consequences in diverse and often dynamic HRC scenarios [4]. The safety
planning process must specifically address the challenges posed by the interaction between
humans and robots. This process includes collision avoidance, safe speed limits, and coordination
of tasks to prevent potential harm to workers and robots. In general, the safety planning process
is notably intricate and requires a significant amount of time, as observed in Saenz’s 2018 survey
[5].
To address this challenge, this paper proposes a conceptual safety planning framework
for HRC. The semi-automatic approach generates possible safety hazards, their causes and
consequences using machine learning and provide them to a human safety expert using a
graphical dashboard. The expert reviews the proposed output, edits it if necessary or adds
additional information. We evaluate our proposed conceptual planning framework using an
exemplary use case that resembles industry needs.
This paper is outlined as follows: Section 2 discusses the current state of the art and its
shortcomings. Subsequently, Section 3 develops a conceptual framework for safety planning
in HRC. Preliminary evaluation results indicating the usefulness of our approach are given in
Section 4. Finally, Section 5 concludes the paper.
2. Related work
Considering the importance safety has in the context of HRC, safety standards and designs that
provide unified requirements and design guidelines for collaborative robotics exist. The work by
Tan provides us with five fundamental safety designs that focuses on robots, work environment,
control systems, monitoring systems, and the application of safety in their construction and
usage [6]. In 2006, Steinfeld proposed the standardization of safety measures [7] and in 2016, the
international organization of safety provided ISO/TS 15066, that specifies safety requirements
for collaborative industrial robot systems and the work environment [8]. Arents et al. [9]
provide a review of studies focusing on safety of HRC systems. 25% of all reviewed studies
did not use any safety actions, and more than 50% did not use any standards to address safety
issues, showing a lack of safety functionalities in HRC.
Wang et al. focus on building a JAVA based safety system with a mathematical model that
takes ISO/TS 15066's restrictions in biomechanical limits and deploys it to an HRC system [10].
In this case, the tests were implemented in the HRC lab at KTH, Sweden. The system uses
psychological characteristic data to provide the operator with a personalized safety strategy
plan. Vicentini proposes a framework that does safety assessment through formal verification
in HRC [11]. It is used for supporting dynamic safety assessment in HRC applications. The
system uses model-based formal verification to explore possible workflows, to identify hazards
and to introduce provisions for risk hazards. It is important to focus that it uses temporal logic
language as it focuses on all details including the irrelevant ones as any type of data could add
value in acknowledging hazardous situations and therefore to mitigate them by introducing
Figure 1: Ontological Structure of an Assembly Sequence Plan
risk reduction measures in the model.
Other research in this area focuses on aspects such as proposing strategies for ensuring safety
and productivity in assembly stations [12], advancing HRC in automotive assembly for safer and
efficient operations [13], improving ergonomics, reducing injury risk in automotive brake disc
assembly [14], and present a disassembly sequence planner for HRC in automating disassembly
tasks [15]. A precise safety analysis for each step of the assembly sequences is still missing.
3. Assembly sequence planning framework
In previous work, we have shown that production sequences for manufacturing tasks can
be generated from model-based specifications of the product and conceptual models of the
production site’s architecture, including its individual Cyber-Physical Production Systems
(CPPS) and their capabilities [16]. Recently, we proposed an ASP algorithm for HRC tasks
[17]. This algorithm generates possible assembly sequences based on the capabilities of the
robot and a CAD-specification of the product. The algorithm allows the extraction of relevant
information for the assembly steps which we can use to support the safety evaluation of the
different assembly steps in this paper.
3.1. Semi-automated safety planning
We implemented a safety planning algorithm generating an ontological description of the
assembly sequence which is shown in Figure 1. Each possible assembly sequence plan includes
a series of n assembly steps. Each step is described by its essential components (elements that
are used in an assembly step like screws, bolts, plastic casings etc.), required actions (such as
joining or gluing), tools (e.g., screwdriver or hammer), and the resources involved (human,
robot or a combination of both).
The safety planning approach follows the safety analysis workflow based on the principles
of Failure Mode and Effects Analysis (FMEA) [18]. Therefore, the information provided by
the ASP are used to identify possible safety hazards, evaluate them, document their causes
and consequences, and estimate their frequency, severity, and risk. For each safety hazard, we
Table 1
Ordinal scale for the frequency, severity and risk class with its description.
Ordinal Scale Frequency Description
1 Rare Extremely unlikely that the event will occur
2 Remote Likely to occur some time
3 Occasional Likely to occur several times
4 Probable Likely to occur often
5 Frequent Likely to be consistently encountered
Ordinal Scale Severity Description
1 No impact Inconsequential, no disruptions
2 Mild Minor impact, manageable
3 Moderate Noticeable impact, requires attention
4 Significant Considerable impact, demands immediate action
5 Severe Critical and urgent, requires immediate action
Ordinal Scale Risk Description
1 No Risk Negligible or no potential harm
2 Mild Risk Minor level of potential risk
3 Risk Moderate level of potential danger
4 High Risk Significant risk of harm
5 Very High Risk Extremely critical situation
identify causes and consequences. A cause refers to the reason behind the occurrence of the
safety hazard while a consequence outlines the potential outcomes resulting from the safety
hazard. The classification of safety hazards is accomplished through frequency, severity, and risk
factors (see Table 1). These values serve as indicators of the importance of each safety hazard
and provide guidance for considering their significance in the overall safety analysis. Risk, in
essence, is the combined effect of severity and frequency. All initial values are proposed by our
prepopulated safety algorithm (PS-Algorithm) and subsequently assessed by the safety expert.
3.2. Conceptual approach for identifying possible safety hazards
The PS-Algorithm consists of four stand-alone models which are arranged in a chain shown
in Figure 3. Each model PS - 1 to PS - 4 is a transformer-based NLP model trained with the
concept of transfer learning. We use existing models and retrain it with specific data gained
from the human safety expert. Instead of using only one model to generate our required output
for the safety analysis, we divided the whole process into four parts:
1. The first model PS-1 uses the concept of NLP to generate the safety hazards as an output
based on the input data which is Component, Action, Tool and the Resource. The
architecture allows converting text information into numerical representations as well as
positional information in the form of a vector. The converted information is processed
via different neural network layers. The output converts the processed data to a text
sequence representing the safety hazard.
2. These safety hazards are used to feed the second NLP model (PS-2) for the generation of
associated causes.
3. PS-3 takes safety hazards from PS-1 as input to form the consequences as output.
4. Our last model (PS-4) is a neural network taking safety hazard (PS-1), cause (PS-2) and
consequence (PS-3) of each assembly step as input. The output layer of the neural network
has three classes, frequency, severity and risk. For each class, a number between one and
five is generated, representing ordinal scale features.
A human safety expert actively contributes expert knowledge as an input for each model.
Over time, the input from the safety expert helps to refine the algorithm’s output. The expert’s
corrections are incorporated into the PS-Algorithm, which undergoes retraining to enhance the
various models (PS-1 to PS-2, as depicted in Figure 3).
4. Preliminary evaluation
Figure 2: Left: Safety Analysis for one assembly step with detailed information about the input sources
at the top and generated possible safety hazards with their assessment results at the bottom. Right:
Safety expert dashboard to edit or add safety hazard information.
We integrated our proposed Safety Analysis (Figure 2) into an existing planning framework
[17] for HRC assembly processes, allowing us to harness all input data we need for our PS-
Algorithm. The data include information about Components, Actions, Tools, and Resources for
each assembly step. This comprehensive approach ensures that safety considerations seamlessly
intertwine with assembly planning in the realm of HRC, enhancing overall process safety and
efficiency. The presented framework is validated on an exemplary collaboration use case where
human and cobot collaborate to assemble DUPLO blocks. We leveraged this information to
automatically generate safety hazards for every step in the ASP, along with their associated
causes and consequences. To achieve this, we utilized ChatGPT API with our models (PS-1 to
PS-3). Our preliminary findings indicate that the output yielded more than ten safety hazards
and up to ten causes and consequences for the DUPLO use case. Moreover, the structured
representation of information has significantly expedited the safety analysis process, eliminating
the need for the human expert to search for required information for each assembly step.
While this initial evaluation showed the advantages of the proposed framework, we have
identified further improvements for future work. In particular, users liked the dashboard as an
easy accessible information source. Although the dashboard currently focuses on representing
the safety hazards and a representation of its causes, relating these to specification models can
improve the analysis of the safety expert in the future. In particular, linking the safety hazards
to process models showing the assembly processes, to development models showing the impact
of design decisions taken, or to conceptual representations of the work pieces, can lead to a
more thorough safety analysis and support better feedback on the found safety hazards.
5. Conclusion and future work
In this paper, we introduce a conceptual safety planning framework for assembly sequences in
an HRC context. The conventional approach to HRC safety analysis involves manual efforts by
human safety experts which proves to be labor-intensive and non-scalable when dealing with
multiple products. To address this, we propose a semi-automated support system that assists
safety experts in analyzing assembly sequences for HRC applications.
Our approach combines expert knowledge with algorithms to partially automate the safety
analysis process, reducing the manual effort and resources required for comprehensive safety
assessment. Moreover, the algorithm’s ability to learn from the safety expert’s input and
preexisting information holds the potential for continuous improvement, leading to enhanced
performance and more accurate safety suggestions over time.
Preliminary results from the implementation of our safety planning framework showcase
significant advantages over manual safety analysis methods. The framework exhibits the
potential to elevate the quality and speed of safety analysis for HRC use cases.
Moving forward, we plan to extend the application of our semi-automated safety analysis to
supports runtime evaluation and re-planning of assembly sequences considering the safety of
the collaboration. To validate the efficacy and practicality of our approach, we intend to apply
the PS-Algorithm to more intricate industry use cases.
Acknowledgments
This research was partly funded by the Bayerische Forschungsstiftung under grant no. AZ-
1512-21. We thank our industry partners Fresenius Medical Care, Wittenstein SE, Uhlmann und
Zacher, DE software & control and Universal Robots for their support.
References
[1] A. Bauer, D. Wollherr, M. Buss, Human–robot collaboration: a survey, Int. Journal of
Humanoid Robotics 5 (2008) 47–66.
[2] B. Sadrfaridpour, H. Saeidi, J. Burke, K. Madathil, Y. Wang, Modeling and control of trust
in human-robot collaborative manufacturing, Robust intelligence and trust in autonomous
systems (2016) 115–141.
[3] F. Vicentini, Terminology in safety of collaborative robotics, Robotics and Computer-
Integrated Manufacturing 63 (2020) 101921.
[4] Z. M. Bi, C. Luo, Z. Miao, B. Zhang, W. Zhang, L. Wang, Safety assurance mechanisms
of collaborative robotic systems in manufacturing, Robotics and Computer-Integrated
Manufacturing 67 (2021) 102022.
[5] J. Saenz, N. Elkmann, O. Gibaru, P. Neto, Survey of methods for design of collaborative
robotics applications-why safety is a barrier to more widespread robotics uptake, in:
Proceedings of the 2018 4th International Conference on Mechatronics and Robotics
Engineering, 2018, pp. 95–101.
[6] J. T. C. Tan, F. Duan, R. Kato, T. Arai, Safety strategy for human–robot collaboration:
Design and development in cellular manufacturing, Advanced Robotics 24 (2010) 839–860.
[7] A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz, M. Goodrich, Common
metrics for human-robot interaction, in: 1st ACM SIGCHI/SIGART Conf. on Human-robot
interaction, 2006, pp. 33–40.
[8] S. ISO, Robots and robotic devices—collaborative robots (iso-15066: 2016), Int. Organization
for Standardization (2016).
[9] J. Arents, V. Abolins, J. Judvaitis, O. Vismanis, A. Oraby, K. Ozols, Human–robot collab-
oration trends and safety aspects: A systematic review, Journal of Sensor and Actuator
Networks 10 (2021) 48.
[10] X. V. Wang, A. Seira, L. Wang, Classification, personalised safety framework and strategy
for human-robot collaboration, in: Int. Conf. on Computers & Industrial Engineering, CIE,
2018.
[11] F. Vicentini, M. Askarpour, M. G. Rossi, D. Mandrioli, Safety assessment of collaborative
robotics through automated formal verification, IEEE Trans. Robotics 36 (2019) 42–61.
[12] G. Michalos, S. Makris, P. Tsarouchi, T. Guasch, D. Kontovrakis, G. Chryssolouris, Design
considerations for safe human-robot collaborative workplaces, Procedia CIrP 37 (2015)
248–253.
[13] G. Michalos, S. Makris, J. Spiliotopoulos, I. Misios, P. Tsarouchi, G. Chryssolouris, Robo-
partner: Seamless human-robot cooperation for intelligent, flexible and safe operations in
the assembly factories of the future, Procedia CIRP 23 (2014) 71–76.
[14] S. Heydaryan, J. Suaza Bedolla, G. Belingardi, Safety design and development of a human-
robot collaboration assembly process in the automotive industry, Applied Sciences 8 (2018)
344.
[15] M.-L. Lee, S. Behdad, X. Liang, M. Zheng, Disassembly sequence planning considering
human-robot collaboration, in: Am. Control Conf., IEEE, 2020, pp. 2438–2443.
[16] M. Daun, J. Brings, P. A. Obe, S. Weiß, B. Böhm, S. Unverdorben, Using view-based
architecture descriptions to aid in automated runtime planning for a smart factory, in:
IEEE Int. Conf. on Software Architecture Companion, IEEE, 2019, pp. 202–209.
[17] F. Schirmer, P. Kranz, C. G. Rose, J. Schmitt, T. Kaupp, Holistic Assembly Planning
Framework for Dynamic Human-Robot Collaboration, The 18th Int. Conf. on Intelligent
Autonomous Systems, Suwon, Korea (2023).
[18] Z. Wu, W. Liu, W. Nie, Literature review and prospect of the development and application of
fmea in manufacturing industry, The Int. Journal of Advanced Manufacturing Technology
112 (2021) 1409–1436.
Figure 3: Workflow of our Prepopulated Safety Algorithm, showing the Human Safety Expert as an
additional input to the common input consisting of Component, Action, Tool and Resource.